Remote access to production infrastructure (death to the VPN)

As I understood the paper, their production version is called überproxy and has access to everything they host, as they moved their applications all on to it (notably, this means SSH now has to go through chrome).

A key reason why BeyondCorp actually works is hardly anybody needs to SSH to prod, and people who do need it, need it rarely. Everything at Google has rich RPC control surfaces and the tools are installed on users' workstations to invoke the RPCs. Status of everything is available via HTTP, in your browser. No need to SSH to a server to read logs or restart a process. Need to collect hardware PMU counters in prod? There's an RPC service for that. Not only do these rich interfaces enable BeyondCorp, they also cut down on insider risk because it's no longer considered "normal" to get an interactive shell session in production.

There are a few. https://github.com/tink-ab/login-service is one I wrote and open sourced when I worked at the associated company. Adding something like https://github.com/dhtech/authservice + https://github.com/dhtech/prodaccess would give you client certificate provisioning and handling.

In the end you'd easily get e.g. daily U2F with monthly cert rotation. Or whatever you end up wanting.

Open source have had this convered for years, but you'll have to look for it.

Disclosure: I'm the main author of all those three projects.

As for handling the TLS part, both Apache and NGINX have the capability to do verification of certificates. The application only needs to parse the headers they pass on to determine the user connecting to the application.

If you don't care about the user (TLS client certs + standard username/password) you can get away with proxying the application through nginx and calling it a day.

Basically you turn on client verification and you're done. If you want to show an error to unauthenticated users, you can make verification optional and add something along the lines of: if ($ssl_client_verify != SUCCESS) { return 403; }

nginx can do the client cert verification for you and pass the results as http headers to the application. so all you have to do in the application is to add some request authentication interceptor that inspects the headers.

You don’t implement it at the Application level. You implement mutual TLS on your proxy and then each application can keep its own auth.

Disclaimer: I haven’t don’t this myself (yet), but have read about it a bit.

You can use openresty and use lua to create a whitelist of certs. Example: https://gist.github.com/phamhongviet/71da4b02bb517000593cf49...

I think most open source stuff supports client certs pretty well, but the issue is getting them to end users. I personally use mTLS as a two part authentication/authorization system; services prove their identity to each other with certificates, humans prove their identity to a proxy server that generates a bearer token for the lifetime of the request. Then each application sees (source application, user) and can make an authorization decision.

I personally use Envoy as the proxy and cert-manager to manage certificates internally. You can peruse my production environment config for my personal projects at https://github.com/jrockway/jrock.us (dunno if that's the real link, my ISP broke routes to github tonight, but it's something like that).

The flow is basically:

1) At application installation time, a cert is provisioned via cert-manager. Each application gets a one-word subject alternate name that is its network identity. The application is configured to use this cert; requiring incoming connections to present a client certificate that validates against the CA, and making outgoing connections with its own certificate. (This integrates nicely with things like Postgres, that expect exactly this sort of setup.) This lets pure service-to-service communication securely validate the other side of the connection. This is nice because, in theory, I don't have to configure each application with a Postgres password, Postgres can just validate the client cert and grant privileges based on that. (I have not set this up yet, however.) I also like the ability to reliably detect misconfiguration; if you misconfigure a DNS record, instead of making requests to the wrong server, the connection just breaks. Saves you from a lot of debugging. And, of course, if the NSA is wiretapping your internal network, they don't get to observe the actual traffic. (But probably compromised your control plane too, so it's all pointless.)

2) The other half is letting things outside of the cluster make requests to things inside the cluster. I use an Envoy proxy in the middle; this terminates the end user's TLS connection, and routes requests to the desired backend, like every HTTPS reverse proxy ever. I wrote a "control plane" that automates most of the mTLS stuff (it's production/ekglue in the repository; ekglue is an open-source project that is agnostic to mTLS, my configuration adds it for my setup). At this point, users outside of the cluster will see a valid jrock.us cert, so they know they've gone to the right site, and applications inside the cluster will see that traffic is coming from the proxy, and can decide how they want to trust that. Right now, everything I run in my cluster just passes through to its native authentication, so it's pretty pointless, but the hook exists for future applications that care.

3) For applications that want a known human user (or human-authorized outside service, think dashboards or webhooks), I wrote an Envoy ext_authz plugin that exchanges cookies or bearer tokens for an internal request-scoped time-limited access token. Applications can then validate this token without calling out to a third-party service, so no latency is introduced. (They do have to be configured to do this, and the state here in the open source world is pretty abysmal. OIDC is helping, and it's trivial to write it into your own application framework. A few applications will just accept an x-remote-user HTTP header, which I found to be adequate, especially if they can trust the proxy with mTLS. Compromising the proxy lets you compromise all upstream apps, though, so I'm looking for a new design.)

I actually wrote this at my last job and don't have the code (it's theirs)... but am slowly rebuilding it in my spare time. Second system syndrome is a bitch. You can follow along at my jsso repository on Github, but it is not ready to be used and I think that most of the stuff I wrote in the design document there is going to change ;)

Anyway, where I'm going with all this is... all the pieces exist to make yourself a secure and reliable production environment. mTLS is pretty straightforward these days, and in addition to the easy route of just doing it yourself, a bunch of frameworks exist to let you get even more security (SPIFFE/Spire, Istio, etc.) For authenticating human users, most of the work has been done in the closed source world; Okta, Duo, Google's Identity Aware Proxy, etc.

It's still sent in the clear and observable by anyone that see your network traffic.

The common assumption is that the company-issued VPN client will only ever be installed on hardware owned and controlled by the company, and never inside a VM.

Realistically, the usual plan is to create controls that are impossible for most non-technical users to bypass, inconvenient for anyone else to bypass, and back them up with the threat of disciplinary action.

If the business prioritizes security, getting to https://news.ycombinator.com would not be possible. If you want to get to https://news.ycombinator.com you use a separate PC.

The PC you use with the VPN is never to be directly connected to the internet. It connects to a piece of dedicated VPN hardware. (could be a Raspberry PI with special software or something far more expensive) That PC can use the VPN, and thus get to various computers within the company, but it can't go elsewhere. No other business is reachable.

The company can allocate IP addresses without NAT and without regard for the rest of the world. There just isn't any connection to the rest of the world, so conflicts can't happen.

It could have to deal with monitoring what is coming and going from the VPN'ed machine. If malware can escape the VPN tunnel, it is less likely to be detected than if it is forced to go through a firewall that is already looking for suspicious traffic. Rather than let a user pick and choose what goes through the VPN and possibly letting the malware make the decision for them, don't let them choose at all. I've worked at places that force you to use the VPN if you aren't connected to the office network. Security of the VPN client aside, it's not a bad idea to force any remote machine to be totally protected from the raw internet.

Most people in industry I have talked to just do VPN for all traffic, even though that seems crazy to me to route your Hacker News traffic through your prod servers' networks (gross). Normally a compensating control would be appropriate, but if you are pursuing FedRAMP Moderate ATO like we did, you can only get away with a pretty small number of alternative implementations, and only Low findings can be easily accepted by the Federal stakeholders. It's tough.

The risk of split tunneling is that the remote client has the ability to relay data in real time. Allowing it means the local printers or laptops on an open WiFi are effectively on your network.

Better bust out the JNCO jeans and Offspring CDs because IPv6 is on it's way and you can bet some deployments will have everything accessible to everything.