HashiCorp Boundary
hashicorp.com
hashicorp.com
I'm excited to see Boundary here! I want to note a few things about Boundary, why we made it, why it is different than other solutions in the space, etc.
* Boundary is free and open source. Similar to when we built Vault, we feel like the solution-space for identity-based security is too commercialized. We want to provide access to this type of security to a broader set of people because we feel it's the right way to think about access control. Note: of course as as a company we plan on commercializing Boundary at some point, but we'll do this similarly to Vault, the major featureset of Boundary will remain free and open source forever.
* Dynamic resource catalogs. Other tools in this space usually require manually maintaining a catalog of servers, databases, applications, etc. We're integrating Boundary closely with Terraform, AWS/GCP/Azure, Kubernetes, etc. to give you live auto-updating catalogs based on tags. (Note: this feature is coming in 0.2, and not in this initial release, but is well planned at this point)
* Dynamic credentials. Existing tools often require static credentials. Boundary 0.1 uses static credentials, too, but we're already working on integrating Boundary with Vault and other systems to provide full end-to-end dynamic credentials. You authenticate with your identity, and instead of reusing the same credentials on the backend, we pull dynamic per-session credentials.
And more! Remember this is a 0.1 release. We have a lot of vision and roadmap laid out for this project and we are hard at work on that now. We're really excited about what's to come here.
Specifically, as a 0.1, Boundary focuses in on layer 3 connections (TCP) with minimal layer 7 awareness for protocols such as SSH. This will be expanded dramatically to support multiple DB protocols, Microsoft Remote Desktop, and more.
Also, we're releasing another new product tomorrow that is more developer-focused, if security is not your cup of tea. Stay tuned.
The Boundary team and I will be around the comments to answer any questions.
Eventually you can migrate towards Boundary nodes (or similar technologies) being the public ingress instead of a VPN endpoint.
(Edit: clarified that I meant firewalls on the end hosts, not on the VPN or elsewhere in the network.)
I hate this corporate speak. You're breaking into the space by giving away (basic, as you will commercialize any advanced) features under the guise of open source altruism. The products HashiCorp sells are open core, and you should be more honest about it (GitLab is!). I wish you operated more like other, real, open source companies that use subscriptions or managed service offerings and don't lock features behind various obscure pricing tiers. This is Shareware 2.0.
The difference between what HashiCorp does and what a real open source company like Rancher does is stark: HashiCorp has products, Rancher builds communities. Contributors to HashiCorps stuff have to play in a very specific sandbox, lest they implement lucrative features. Contributors to Rancher help the community at large and have full visibility into the codebase, empowering them to fix or add functionality without restrictions.
Edit: 0: It's (presumably) technically not false now, but the implication is that $THING is honestly intended to be FOSS, immediately followed by admiting that their actual intent is to sabotage that embrace-extend-extinguish-style as soon as it's commercially expedient to do so.
Sabotage??? Wow, that’s quite an accusation for a company that’s, you know, a company. You might have an argument if they kept quiet about plans to monetize the product later, but that allegation is laughable.
If you’re not comfortable with the terms, don’t use the product. They’re being upfront about their plans. This anti-commercial position is hypocritical.
"I want all of the functionality I want without having to pay for it." I hate how discussions around software businesses so often descend into purity tests around how much a company chooses to give away. Software is indeed eating the world, but the eternal battle of who has to pay for the underlying tools of said software continues.
Boundary is free and open source. There is no corporate speak here. It is FOSS licensed (MPL2) and everything announced today is completely FOSS.
We do sell open core software and if there is any place where you feel we aren't being honest about that please let me know and I'll work to address that. I added that "NOTE" at the end of the point specifically to ensure I was being honest and show I wasn't trying to hide anything.
We are also starting to offer managed services for folks who prefer to consume our software that way. The managed service offerings do unlock the typically enterprise features. Example: https://www.hashicorp.com/blog/hcp-consul-public-beta
Is HashiCorp known to do this?
All I've heard are good things about HashiCorp from people who use HashiCorp products.
Second, it can't be forgotten these are companies. A company exists to create value for itself in some way.
It's the natural behavior of any company.
However in my opinion, "open core" design seems to be very very preferable amongst technologists (myself included). Essentially we are paying for additional features which normally we'd wait years from a sole contributor.
I think you might be confusing vault with another product?
We self-host vault in production, and it doesn't cost us a dime.
(other than the engineers we pay internally to operate it, of course)
HashiCorp and other companies doing "devops" tools are known for using "open core" and hijacking the spirit of open source in many ways.
All three have wildly different values and historically corporations aren't very good at listening to anyone that isn't waving a check. They use reasoning like "priorities" to close source formerly open source projects, bend project values to reflect their own values, and wedge projects with funding in exchange for representation or control. Corporate controlled and born projects are often used as marketing or for good PR, a cursory browsing of a company's Twitter page will show how they utilize it for this type of end.
I don't really read Mitchell's speak as corporate or double speak, but I do think that referring to HashiCorp (and other) projects as "open source" is a half truth. The line that I draw here is that I don't think Mitchell is lying, rather, I think that open source is now an umbrella term that means very little and really terms like open core, free and open source software, etc are more concise. We owe that outcome to inviting our corporate friends into the fold of open source with not enough restrictions, tracking, and accountability but there's a piece of me that feels this outcome was largely intentional because it's become a means to an end as I described above. These could just be feelings but the situation is common enough that it's relatable.
I'd encourage corporations to be more transparent in their verbiage, their investments, and their representation in these projects so that it doesn't continue to confuse people who participate in and enjoy the "free" side of open source. When I look at an open source project I'd love to know if a majority of the maintainers or funding comes from a corporation. If those things are true, then as someone who highly believes in the ideals of free software I may want to stay far away from people who are susceptible to corporate influence and values. On the other hand, that increased transparency may help clear the air and prevent issues from being perceived as non-transparent or outright misrepresentation.
Spot on. Corporate "open source" is often open only in terms of licensing, but not in terms of values.
Many companies use tricks to prevent successful forks and keep tight control over the development process.
This indirection provides a way to keep your public and private (or even private and private) networks distinct to remove "being on the same network" as a sufficient credential for access. At the same time, it ensures that the traffic is only proxied if that particular session is authenticated.
And could this be done in a way that is completely transparent to the user (without them having to start a connection to the worker first, and then make a connection to the desired service)?
The use case I'm thinking of is for IoT or robotics, where you have devices you want to manage being deployed into remote networks that you don't have much control over. It's really helpful in this situation if devices make outbound connections only, so that network operators don't have to configure their firewalls to port forward or set up a VPN.
Edit: clearer language
CloudFlare Access allows this, using the cloudflared daemon, which acts as a reverse proxy. It essentially means the endpoint can be closed off to incoming connections from the internet, and you don't need to maintain various firewall whitelist (and hope they don't go out of sync)
Is something like this on the roadmap for Boundary?
As you can imagine we did quite a bit of research with our existing users/customers while working on the design of Boundary. One thing we heard almost universally was "please don't require us to install another agent on our boxes". So we decided to focus initially on transparent use cases that only require running additional nodes (Boundary controller/worker) without requiring additional software to be installed/secured/maintained on your end hosts.
> the endpoint can be closed off to incoming connections from the internet, and you don't need to maintain various firewall whitelist
If you think about this a bit differently, a Boundary worker is also acting as a reverse proxy gating access to your non-public network resources. You can definitely use Boundary right now to take resources you have on the public Internet, put them in a private-only subnet or security group, and then use a Boundary worker to gate access. It's simply a reverse proxy running on a different host rather than one running on the end host. You wouldn't _need_ to add a firewall to ensure that only Boundary workers can make incoming calls to the end hosts, it's simply defense in depth.
I did basically the same thing years ago for remote console devices deployed inside various customer networks where I had little or no control over the network. At that time, I used OpenVPN to automatically connect back to our "VPN servers" -- providing access to the device even if it was behind two or three layers of NAT (which, unfortunately, wasn't uncommon!).
But I had a chuckle at the idea of you wheeling a whiteboard into your house (if that is where it is filmed).
Can you view logs of SSH sessions after the fact?
Can you live-view a session?
Can you require a pairing authorization like with https://github.com/square/sudo_pair?
Our initial focus is on making the connections easy. We have some work to do there still. We'll then move on to more management features like this. They're both super important but from an initial adoption perspective we feel the latter is moot if the former (connections) don't work easily.
I've been thinking about that the past couple of years, with today's building blocks an SSH alternative is so easy to build. I bet if you guys were to build or back such a system it would be the right quality and get the adoption it needs.
Opaque SSH sessions are such a thorn in my side.
1. It's not clear to me how you actually secure the targets? Do you just enable access to the IP address of the controller proxy? In the video you mention a gateway but there's no description of that in the docs?
2. Is it possible to proxy a web browser session? Or is it limited to individual requests via something like curl at the moment?
Zero Trust means authenticating per application instead of per network. For more context see https://about.gitlab.com/blog/2019/04/01/evolution-of-zero-t...
Proxying connections as Boundary does seems like the most elegant solution to achieve this in a way that doesn't require modifying the application.
E.g, if I want a Consul backed Vault, whilst using Vault to generate TLS certs or other creds for Consul. Especially if I want to run either/both of those services using Nomad, backed by Consul. Hopefully I wont have the option of authenticating against any of these services using Boundary. Especially if Boundary is backed by Consul.
One way we're simplifying this a lot for people is the introduction of our managed services[2][3]. We understand not everyone can use a managed service though!
Boundary will integrate fairly deeply with Consul/Vault but these integrations will be optional.
[1]: https://www.vaultproject.io/docs/configuration/storage/raft [2]: https://www.hashicorp.com/blog/hcp-consul-public-beta [3]: https://www.hashicorp.com/blog/vault-on-the-hashicorp-cloud-...
That said, I'm moving to a central distributed Vault that is mostly going to exist as a PKI so I'll only really need to repeat this process once more! Going to be using the raft internal engine for this one, and spread it physically across the globe so performance is going to be pretty terrible by design, but it should be quite resilient!
In our case, we use lets encrypt to get certificates for vault and then bootstrap a vault cluster with internal storage. Then you have vault and you can use terraform to configure a consul TLS backend.
And then there is a little hitch, because consul-template cannot easily create multiple files from a single vault API call, so you cannot use consul-template directly to create the necessary certificate files. We've written a small messy tool there. But once you have that, it's fairly straight forward to generate consul + nomad TLS certs for the trust and then you're set.
When I first looked at the product description, I thought I might be looking at a "zero-trust identity-aware-proxy" sort of thing, but as I read more I got more of the "privileged access management" vibe with more of a focus on controlling access to infrastructure for developers vs. applications for end users.
* Azure App Proxy
* Google IAP
* Amazon WorkLink
* Cloudflare Access
* Zscaler Private Access
* Duo Beyond
* Hashicorp Beyond
We provide a lean PAM solution for multi-cloud infrastructure access.
One example. I have been testing smallstep, which puts IDP around ssh (with group management), and also includes a dynamic host catalog (hosts run an agent that phones home to your identity provider).
However, I am very excited about Boundary as it seems to be a much more comprehensive solution.
- Full HashiCorp stack (Nomad, Consul, Vault, Terraform)
- Cadence (https://temporal.io/)
- Microservice architecture over gRPC and Consul Connect
- All services written in Go
- Customer clusters are created/managed by programmatically running Terraform using just-in-time cloud credentials from Vault
- All internal TLS certs for customer clusters dynamically created using Vault
- All external TLS certs for customer clusters dynamically created using LetsEncrypt via Terraform
- Frontend is EmberI have soooo many questions about best practices doing this. I run a service that needs to dynamically provision AWS resources, and lacking a clear path to do this programmatically, I shell out to Terraform.
* I assume you aren't shelling out :). Do you have any additional helper libraries on top of the Terraform code base to make it more of a a programmatically consumable API, as apposed to an end user application?
* Are you still pointing at a directory with resources defined in HCL, or are the resources defined programmatically?
* What are you using for state storage?
* What is the execution environment for the programmatic Terraform process? Since Terraform uses external processes for plugins, I've hit some issues with resource constraints around the max number of process sysctl's in containerized environment where I have multiple Terraform processes running in the same container.
edit: formatting
Answering your questions:
> * I assume you aren't shelling out :). Do you have any additional helper libraries on top of the Terraform code base to make it more of a a programmatically consumable API, as apposed to an end user application?
We in fact are. There are lots of security concerns you have to consider with this. We published a library to make this easier: https://github.com/hashicorp/terraform-exec
> * Are you still pointing at a directory with resources defined in HCL, or are the resources defined programmatically?
HCL mixed with the JSON flavor of HCL for programmatically generated stuff. Variables in JSON format also programmatically generated.
> * What are you using for state storage?
We output it to a file and handle this in an HCP microservice. We encrypt it using the customer-specific key with Vault and store it in a bucket that only the customer-specific credential has access to. If there is an RCE exploit somehow in our workflows, they can only access that customer's metadata.
> * What is the execution environment for the programmatic Terraform process? Since Terraform uses external processes for plugins, I've hit some issues with resource constraints around the max number of process sysctl's in containerized environment where I have multiple Terraform processes running in the same container.
Containers in HCP and VMs in Terraform Cloud due to increased isolation requirements. HCP has less strict requirements because the Terraform configs and inputs are more tightly controlled.
>>
> We in fact are.
Words can not express the joy I feel in reading this. Thanks so much for the responses!
I'm former creator and tech lead of Cadence and currently tech lead of Temporal.
This might trigger some folks but have you explored any options for delivering some or all of the Boundary infrastructure through serverless/faas?
A desktop client with a list of services/targets would also be great. Especially for the less technologically inclined individuals.
I know that people have their own opinions on port knocking but I find it as a good tool to remove a lot of noise, some pre built tool for that would be nice but could always just use fwknop-2
A desktop client is on the way, we already have an internal build of parts of it but it requires more work and didn't make it for 0.1.
boundary proxy is an ok step but user experience should be streamlined especially if it's for teams and orgs and not just individuals who want to hack scripts but I full understand it's a 0.1 release.
Another thing I couldn't find in the docs is support for multiple installations, let's say I have different vpcs (In different accounts) or I have one on-prem installation and one in a cloud how do I login/switch/configure the cli to work seamlessly with multiple controllers.
I did a quick search in the GitHub repo for WireGuard and didn't get any results so I guess you aren't using it.
Thanks a lot for the great products, but please give us managed Nomad already. Or even better: a Heroku like app platform. I want to give you money, but I really dislike your companies' enterprise offerings.
BTW I believe there's a great opportunity for Hashicorp right now. Cloud providers are good at selling building blocks, but are terrible at selling a vision of how you should build your applications. On the other hand, low code / enterprise application platforms are a disgrace as always. IMO a coherent stack of managed Nomad + Consul + Vault could provide a solid middle ground for those who want to build apps without the burden of managing K8s or navigating through the incomprehensible maze of products offered by public clouds.
(1) HCP Nomad is coming. We announced HCP Consul public beta and HCP Vault private beta today (on AWS, more clouds later). HCP Nomad is planned but not quite ready to talk about beyond that yet. That is "managed Nomad."
(2) Re: Heroku-like app platform. Watch tomorrow's keynote or catch up on our announcements tomorrow. It isn't this, but I think it'll give you an idea of the vision we're heading towards and that is relevant to this idea.
So Hashiku? :)
Seriously Heroku seems to have stopped innovating. I wonder how much is Heroku worth now.
Let’s say you have an org that’s doing the whole Consul/Nomad/Vault thing, and starting to have their Nomad jobs using Consul Connect (and it’s proxies/gateways for external).. that’s already a proxy sidecar used for all service ports. How does Boundary fit here? Is it put before/after Connect, is the plan to integrate them, or are they supposed to not be used together?
We'll be looking more closely at other integration possibilities going forward!
I'm sorry, but please cut the corporate-speak.
Reality is that your statements are different from your actions.
"similarly to Vault, the major featureset of Boundary will remain free"
Sounds great doesn't it.
Except Hashicorp decide to hide Multi-factor authentication in Vault behind the paywall.
I mean, I'll forgive you putting a lot of the Vault features behind the paywall (e.g. replication).
But for a security product. Putting a core component of 21st century security (MFA) behind the paywall ?
Pretty unforgivable.
This is not something new. The earliest open source project that I can recall is https://github.com/bitly/oauth2_proxy (albeit it might be missing the part where proxy passing identity to the backend).
Pomerium is another open source project that's actively maintained. I've been using it as a reverse proxy to all my homelab websites (grafana, miniflux etc). I can now safely access all of these internal resources from outside of my home WiFi with automated SSL certificate configuration and renewal.
You can theoretically protect your SSH connection via these IAP proxies, using the Chrome SSH extension and open source SSH relay implementation like https://github.com/zyclonite/nassh-relay (but I personally haven't tried that).
Disclaimer: I work for Google and am a casual contributor to the Pomerium project.
[0] https://gravitational.com/teleport/
Disclaimer: I have no affiliation with any of these companies.
I think moving away from VPN's is gaining more adoption and a good thing overall.
https://github.com/oauth2-proxy/oauth2-proxy is a maintained fork.
The old distinction of "internal network" and "external network" doesn't make much sense.
It's a BeyondCorp like a user identity and layer 7 aware access proxy for RDP, SSH, Web, and Database protocols with privileged access management, native two-factor auth agents, and device trust policies.
Disclaimer: I am a core maintainer of this project.
By default I expect google to try to lock me in the GCP and do not trust their OSS tools
Expect consolidation. That or it becomes a commodity expectation of any other purchase, and not a selling point.
Lightyears ahead of teleport or any of the other solutions out there. Built for great auditing and zero trust.
Best of all it’s multi-protocol. So you can do SSH, SQL, K8s, HTTP all with one access system.
Had it in prod for almost two years. Gonna be a long time before hashicorp or anyone else can catch up with the level of depth.
I haven't found a conclusive answer in their documentation yet.
Teleport is SSH based so you can tunnel other protocols.
(disclosure: small Tailscale investor)
* wireguard (faster) * easier * more stable
- Better throughput overall.
- better NAT holepunching. E.g. ZeroTier gives up entirely with "symmetric NAT" where each outbound connection gets a random source port, but Tailscale has a few extra tricks that it can try (including opening a whole bunch of outbound connections, trying ports at random, and hoping the birthday paradox will kick in, which I think is pretty cool.)
- But most of all, Tailscale didn't suffer from weird intermittent throughput/latency issues between different cloud providers the way that ZeroTier did. Sometimes my machines could talk to each other pretty fast, other times it was clamped down to ~10 MB/s for no apparent reason. Sometimes it only showed up in one direction, sometimes both. I gave up on trying to troubleshoot it when I discovered Tailscale.
That said, I still like ZeroTier a lot and think it's a great project. It also provides a whole LAN layer, with stuff like actual broadcast traffic, for which Tailscale has no equivalent.
Boundary sounds like the perfect mash-up of Google's bastion-less SSH access to GCE instances and actual IAM. Exciting!
Is this the main idea behind BeyondCorp and CloudFlare One, as well? If so this is the clearest explanation I've seen of it.
If you're looking for something like a VPN and you're just going to SSH over it, either would probably work for you, but while Boundary can allow users to only connect to port 22 on certain hosts, I think if you wanted to do similar with Tailscale you'd be in iptables/ufw and "tagging / authz-ing traffic with unix uids" territory.
Boundary comes with built-in wrappers for ssh, rdp, and postgres, but you can "boundary exec" to run some other application inside the TCP-wrapped transport, apparently.
edit: seems nothing that complicated, more like ssh-style tunnel where Boundary has a local listening socket which you need to point the client to. That is if I'm understanding it correctly.
https://github.com/99designs/aws-vault/issues/578 was for an issue with remote servers accessing the localhost ec2 metadata service that aws-vault can run, that worked exactly by using DNS rebinding. It was fixed only months ago, so it seems like this is a developing area and if I were on a red team or pen testing, I would play around with more.
I visualize the "localhost hole" problem of blindly trusting localhost as an air gap in a pipe (like [0]); anybody could come along and either drop poison in the pipe, or redirect the water coming from the top to their own bucket, or both.
I appreciate that Boundary gives completely generic identity-aware-authenticated TCP sockets, but I don't know of a way, today, to make those not accessible to browsers through dns rebinding attacks.
This is probably much much too far in the weeds and this is unlikely to contribute to a major breach (unlike the aws-vault one where of course attackers would try to access the fake metadata service on the default port, because it's high-value and on a well-known port), but I'm interested in the space.
[0] https://news.ycombinator.com/item?id=23265509 [1] https://districtsales.ca/wp-content/uploads/2019/07/tru-gap-...
e.g. something like a docker-compose sidecar exposing an nginx container to users via boundary would really help me understand how this is supposed to be used in practice.
Looking for an example like my comparison here between argo, wireguard, tailscale, letsencrypt, caddy, and ssh ingress: https://gist.github.com/pirate/1996d3ed6c5872b1b7afded250772...
Would I just have Boundary authenticate via Cloudflare Access and whatever identity provider Cloudflare One is integrated with and move to the RBAC policy phase of the authentication - is this where i am seeing that additional value from Boundary by having that additional on demand credential rotation to various internal Apps and DBs once i am past the SSO stage? CF One is more of a vertically integrated all in one service addressing all of my other networking and security needs so it's not really going anywhere.
I think you guys could do well to release a Cloudflare integration paper, it might help with traction on on-boarding customers.
Thanks!
link wasn't working for me, so.
These "non-VPN" solution seem to use a client on your machine that change any DNS lookup through the OS layer by hooking into gethostaddr() and returning the same IP for all domains if they are in the list of hosts that should be virtualized. Then only the traffic to domains that are needed is virtualized, anything else is untouched. YouTube and Netflix won't get piped over your company network, as an example.
Disclaimer: I don't really know that this is how it works but this is how other providers do it.
Armon mentions Okta and Ping, does anyone have any recommendations in this space that would work for managing a small team with occasional on/off boarding of contractors?
Would it make sense to use boundary as a way to manage access to web-based developer environments using an IDP for authentication (e.g. Github, Google, Okta, etc). I'm thinking of tools like JupyterLab/Hub, RStudio Server, etc. Or is that outside the intended scope of Boundary?
Way too much ceremony for scientific compute sites tho