Hacker News Reader

I roll my eyes so hard at people who say "well our network is secured, so it can insecure on the inside" I'm surprised I don't have a repetitive stress injury by now. It shows a shocking lack of foresight and is honestly insanely unprofessional. This is the exact same kind of opinions that lead to just about every major data breach in recent memory.

Secure everything, and assume that everything is compromised. Anything less is downright negligent.

With all due respect, this is like saying the more locks you have, the more secure your building is. I mean, yes you have to lock some doors to have security, but on the other hand putting a lock on the bathroom door, the light switch and the flush handle isn't increasing security, and could be reducing security. I feel somewhat the opposite -- there should be a very good reason to add security overhead, and without a compelling story the default should be open.

This is why i've been so happy reading about wireguard in the kernel recently. This makes it possible to have a very secure, low hassle default that is completely independent of applications.

> putting a lock on the bathroom door, the light switch and the flush handle isn't increasing security

It certainly is if the thing you're securing is the flush handle itself.

Like all security questions, understanding exactly what you need/want to secure and the threat models surrounding it is extremely important.

We put valuables in a safe even if the door locks.

The key work is "valuables". You probably wouldn't put your pants in a safe (though I acknowledge that depends on your roommates). I'm saying that going through the motions doesn't make something secure. How many passwords are in slack channels or github repos? Creating a password barrier, but neglecting a secure mechanism for rotating them and distributing them doesn't buy any security.

The problem with this analogy is that its not a good one. We're humans so we tend to try and explain things in terms of concepts we understand but often latch on to incorrect ones.

The problem with locking every object in your home isn't that its an inherently bad thing, its that it introduces too much overhead to be worth the hassle. Now, if there were some touch ID enabled force-field that unlocked things only for you, within milliseconds and was highly reliable to work, that might change the dynamics a lot.

And that is what happens with crytographic security. There is very little overhead that modern systems impose on securing your systems so there is no reason for you to not enable them (setup may be hard, I acknowledge its a problem but its a one time cost to pay).

Taking the analogy further; if you had to repeat the authentication flow for every digital object it would still introduce a lot of hassle. So digital cryptography gets around this by instead requiring short lived tokens so the cost of enabling cryptography is amortized across all your digital assets. In our analogy; it would be that every object in our home is locked, but we use faceID to get a short lived key that is cached for e.g. 10 minutes. All locked objects check for the existence of that short lived key.

If you take protecting your users seriously then any of their personal info counts as a "valuable" so yes, you should secure everything. And it strains credulity that you might be running any kind of actual business where it would be appropriate not to take protecting your users' data seriously. Even if you're doing something as mundane as hosting cat pictures, you have email addresses and password hashes and analytics data.

See, I do infrastructure work in companies that often don't even expose their services. They just want a database and a frontend for a very small internal app that will never get out. And yet, a lot of the tools are designed "for the web" and, rightfully so, enforce https, ssl, certs, you name it.

Now the issue is that they are a real PITA to implement in some of those networks, where you have to use tons of private registries and repositories, and to get everything working with certificates. Because those companies are not used to issue certs, often times it take them months to do so, as they have a very rigorous process, so everyone wonders why this tiny app never goes to "production" and why we are blocked by such restrictions.

Now I'm not even accounting for the fact that you sometimes actually need to go online to install some of those stuff. Recent example was node-sass, dependancy from a composer package, which you would expect to just install from the composer registry ; but no, it has a setup script that goes to nodejs.org or something, which of course you cannot fake because it will have to check certs as well. So again, something designed for the web that'll never work seemlessly elsewhere. (There are workaround of course but I'd love to spend my time elsewhere).

To give you an idea, the cloud I'm working on right now is pretty much entirely offline, for both egress and ingress. There is a nexus mirror-proxy to dockerhub and that's it, even this is in the LAN. So really, having to configure cockroachdb certs (or anything else for that matter, like etcd...) is useless, time consuming, frustrating, and counterproductive.

> very small internal app

Is cockroachdb the right DB for this use case, though? A more traditional Postgres/MySQL or evening sqlite (depending on needs) is often plenty good for a small internal app without requiring this extra security

> the cloud I’m working on right now is pretty much entirely offline

I fee like “cloud” has lost all meaning it may have once had

> Is cockroachdb the right DB for this use case, though? A more traditional Postgres/MySQL or evening sqlite (depending on needs) is often plenty good for a small internal app without requiring this extra security

It's not, but the app I described is just an example. Some use cases involve bigger dataset and resilience.

It's not cockroachdb specifically but any of those "web" tools that, used internally in a closed environment, become a PITA to manage. Language tooling, repositories, dbs... Etcd for example is used as an internal storage for tenants (for example to store DNS information) and the team that manages it cannot upgrade right now because they have to figure out first how to implement ssl.

> I fee like “cloud” has lost all meaning it may have once had

Well, how would you call an on-prem install of OpenStack ? It's their internal cloud. Cloud has not much to do with the web or with the fact that it is publicly exposed, it's a tool to manage infrastructure. And to be perfectly honest, "cloud" isn't a proper term to begin with. Years ago when the word came out one of my professor said they basically renamed "grid" to "cluster" then "cloud". They used to administer hundreds+ of machines and multiples of that of jobs using pssh. They didn't wait for kube, and they didn't need ssl between each and every API and services.

Now don't get me wrong, the public cloud provides a great service and tooling coming out now should be web ready and web safe. Just, please, pretty please, give us a --insecure and --offline even if it means having it all over my ansible code, so we can get the job done without having to spend my days working around those tools.

That sounds reasonable, thanks for the detailed response.

> And to be perfectly honest, "cloud" isn't a proper term to begin with.

Completely agree, it’s just a marketing buzzword.

> Just, please, pretty please, give us a --insecure

Have you taken a look at [0]? (All caveats around running an insecure cluster apply)

[0]: https://www.cockroachlabs.com/docs/stable/deploy-cockroachdb...

the built in resilience via N nodes of something like CRDB can be useful even with super-small-scale data/traffic

You don't need a publicly signed certificate if the services are only used internally. It's not like you want people to look at your database with a browser anyway. All you need to is create a private CA and then two private keys/certificates (one for server, one for client) sign it with the CA and add the server certificate to the database, add the client certificate to your web server that accesses the database and finally add the self signed CA to both database and webserver.

There is no cryptographic security benefit of publicly vs privately signed certificates. After all publicly signed certificates are just privately signed certificates whose CA has been added to your CA store. So just add your own CA to whatever program is needed (do not add your own CA to the whole system). The annoyance is keeping those certificates and CAs up to date. Tools like Hashicorp Vault can automatically generate certificates for you if you want to automate this part.

I came here to mention this, too. It's been shown from numerous leaks that defense in depth is a much more effective approach than simply relying on network security.

You mention that query response data can be sniffed by an attacker in case networks are compromised. The much worse scenario is that the passwords themselves can be intercepted, giving attackers full access.

Relying on network security is usually a bad idea even if you don't have defense in depth. (Ie it's less bad to rely on some other one thing)

All of our internal communication already is secured via wireguard and communication can only flow through the wireguard interface. Does adding cockroach's own encryption on top of this add anything?

I am unfamiliar with how setups like these usually work, but it could an active attacker MITM your network?

You wouldn't be able to decrypt the wireguard packets unless you had each party's private keys

What happens when someone compromises your wireguard overlay network?

The cert business is not just about encryption, it's also about authentication over an untrusted network.

Not much if it’s set up in a standard way because each node will only communicate directly with the node it’s expecting too. The communications are secured with the other node’s public key.

So if you have 10 boxes interconnected in a full mesh, and one of those boxes gets compromised, then it's all fine even if you have been relying on the wireguard vpn being secure?

CockroachDB annoyingly requires manual certificate creation and distribution to each node, and if you don't set those up then you can't use any authorization at all. It's strange and user-hostile to disable auth completely just because you don't use certs.

Also some environments like Kubernetes might already have network encryption (provided by a service mesh) and don't need another layer.

Certificates serve both encryption and authentication, but encryption can be done with self-signed certs automatically generated by the nodes upon joining the cluster and authentication can be handled in the joining process by using a token or some other bootstrap value that's easier to manage.

> CockroachDB annoyingly requires manual certificate creation and distribution to each node, and if you don't set those up then you can't use any authorization at all. It's strange and user-hostile to disable auth completely just because you don't use certs.

This comment is not any more true as of CockroachDB 20.1 (Upcoming in Q2 2020).

This version will enable you to set up your own transport-level security with clients using password authentication to crdb.

One part where it makes sense to me is that CRDB often talks about deploying their product in K8S as a good idea.

In that scenario, it's very easy to end up in a situation where CRDB communications are encrypted twice, with separate certificate management systems. Not just the client->db connections, but also the node->node ones. Especially with things like Istio becoming more popular.

Yep, and besides these concerns, there's the difficulty in modeling the trust relationships and dependencies in this kind of "secure internal network" world view. The relationships are complex and you have low likelihood of your mental model matching the reality, because the relationships are not explicitly configured.

I actually have a similar concern. In a previous job I tried to push for internal services communicating over TLS with a custom certificate authority. It was ultimately not adopted as too much complexity, including operationally when something goes wrong the developer should always be able to issue unencrypted requests from an internal network. It seemed to boil down to removing as many hoops as possible when something terrible happens and automated tools fail and manual recovery is needed.

I am a huge fan of mTLS but it is quite difficult to set up and I'm sort of taking a "wait and see" approach, especially with Google pushing ALTS, which sounds pretty sane. My dream is that you can ignore "VPCs" (i.e. trusted networking, firewalls, etc.) and let applications decide whether or not to allow traffic. "You have presented an incorrect TLS certificate, no response for you."

The problem I have is that I'm not sure what method you want to use to inject certificates. The current systems seem to be "something will figure it out for you at runtime, and your application will be none the wiser". (This is things like Istio and Linkerd.) I am not sure I'm convinced this adds much security. Anyone that can talk to the apiserver that injects certificates can probably convince the system to give you a certificate for your rogue container. And, I'm not sure that applications have enough visibility into the networking layer to really add any trust. I'd like for each incoming request to have some provenance information attached with dimensions like "this container was built from trusted source code and each commit that it contains was code reviewed", "this container was launched from a manifest written and applied by a trusted engineer from a trusted machine", and of course "this request was authorized by a human user whose data is being accessed". Right now, I feel like you have to hand-wave to get the first two. Sure, maybe Istio thinks the container is legit, perhaps because it asked a container notary (very early stages of support), and so now it can open a TLS connection to anything in the cluster. And, adding a request-scoped JWT is easy enough to provide the "here is the human user that made the request". I just don't think you can hide this from the application with a service mesh; every application needs to be aware of what's going on so it can say no to suspicious requests. And for that, I don't think we're really there yet. We had this at Google and it worked well, but I haven't found anything that works well in the outside world yet.

With that in mind, I totally understand why people trust the "well, it's all behind a firewall" model. It's not very good, but it is easy. I hope that in the next year or two we can get to where I want to be, but I'm also not sure that anyone cares. "Good is the enemy of great" and a private network seems to fit people's mental model pretty well. The only thing that automatic mTLS gets you right now is protection against "SSL added and removed here ;-)", and it's only the NSA that's really doing that. If you aren't a target of the NSA, your data is being exfiltrated by employees with access, or by simple application level bugs (XSRF, SQL injection).

This is all very rambly and I'm not sure where I'm going with this... but I wish that someone made application-level transport security work and it was trivial to set up.

> Anyone that can talk to the apiserver that injects certificates can probably convince the system to give you a certificate for your rogue container.

This is pretty infrastructure-heavy but I think my ideal solution would be to have a trusted orchestrator service on every machine which has its own certificate and accepts human-signed build artifacts to be run in a new container on that machine. It could verify the signature of each uploaded artifact and send CSRs for them to a hardened signing server, which returns the certs to be mounted (or sent over a standard initialization API) into the new containers.

Your options for compromising this are:

1. Get a malicious build artifact signed and submit it to a machine for execution. This shouldn't be possible without compromising an actual developer's credentials to sign the artifact.

2. Send your own custom CSR to the signing server to get a signed certificate. This shouldn't be possible without compromising the certificate from one of the orchestrator services to sign the CSR.

I think this should work as long as you can guarantee that services can't break out of their containers and as long as there's some hardware root of trust ensuring that your orchestrator service is genuine and the only thing that can read its CSR-signing certificate.