The MongoDB hack and the importance of secure defaults

Even knifes are sold with some package that prevents them from cutting before the package is removed.

I agree, I don't think your job is done just because you wrote somewhere "pay attention to this".

This is not exclusive to DevOps, though DevOps is a hot buzzword right now so lots of posers are flooding in. Most people, no matter their station or the prestige of their company, have no idea what they're doing. Knowing this is one of the most important things for dealing with the corporate world (I think I'm missing a couple of the other most important things, though...).

Different people try to cover this up in different ways. One popular way is acting like they always know about new cutting-edge tools sponsored by AppAmaGooBookSoft and that the next thing will finally be the Silver Bullet we've all been waiting for.

This impulse has brought us the prominence of node.js, MongoDB, the proliferation of "big data", and many other vastly-overdeployed niche products that have been ravenously and shamelessly misused by incompetent people trying to fake it through their careers. Our standards for this are, frankly, sad. It must have to do with a combination of a cultural bias against non-junior engineers outside of Java/.NET-land and putting completely unqualified MBAs in charge of tech (and yes, this is also applicable in startup world via VC proxies) -- but I digress.

Been hearing that song and dance re: docker and k8s for the last year at least, and boy is it ever tiring. Docker and k8s are both very niche, very immature products that greatly complicate system administration. They are missing features that you rely on and that you will have implement horrifying hacks to work around. Why are you doing this to yourself, again? Oh, because AWS is too expensive and you want to consolidate (fake reason btw, real reason "because it makes people think I'm smart")? Yeah, about that...

It's fun to throw together a lab for experimentation, maybe hook up a weekend project, but no sane Real Company is going to be moving its stuff to all-containerized k8s any time in the next 2 years.

My current project at work? Converting all of our environments to docker/k8s...

Furthermore, running "apt-get install" usually isn't enough to get something exposed onto the public network, even for home users, whose routers block all incoming ports by default. Someone has to go in and explicitly open the traffic before something like Mongo gets exposed; there's no reason that labs, developer machines, etc., would be publicly accessible.

Mongo is hardly alone here; comparable services like MySQL, PostgreSQL, and Redis also install without a password by default. For MySQL, this is perhaps less noticeable because Debian/Ubuntu (and probably some other distros) include a prompt to allow the user to set a new root password as a post-install script. For PostgreSQL, the default mode uses IDENT authentication, i.e., it's accessed by dropping to the "postgres" user ... and, like all other system users, including root, this user is passwordless by default.

For Redis, it's passwordless by default and I'm sure there are many installations that have this same "vulnerability" in redis-land (there are also many installations that have real vulnerabilities, because Ubuntu's redis-server package in 14.04 hasn't been patched), because people rarely think about redis auth either. Just since last year, there is a feature called "protected mode" that requires the user to explicitly disable a flag if they intend to run a server that is a) bound to a non-loopback interface and b) not passworded (but it doesn't actually require someone to set a password). That's a cool feature, but as users don't RTFM, I'm sure they either flip protected mode to false without understanding what it does or just give up on redis and install MongoDB instead. :)

The point here is that pretty much everything you can install on a server is passwordless by default. I don't think the system of someone who never thinks about passwords would last very long at all. This is NOT a MongoDB-specific anomaly. Passwordless is the typical case for most services.

A feature like Redis's protected mode is a nice bonus, but it's not fair to lay responsibility for this at the feet of Mongo or to call it a "vulnerability", as the OP does. The responsibility lies with people who say "I'm going to put all of my company's data in here and push it live without thinking even a tiny bit about security."

Do a lot of people do that? Yes. Should MongoDB take more steps to protect such people from themselves? Maybe. But this is a very basic, very routine thing, which is easy to look up and rectify -- not some obscure anomaly buried under 600 pages of documentation or requiring a special compile-time flag. Mongo doesn't deserve the heat for people who never set up a password.

But most of us chose not to do that for a living.

- [1] https://www.elastic.co/guide/en/elasticsearch/reference/2.0/...

- [2] https://www.elastic.co/guide/en/elasticsearch/plugins/2.2/di...

- [3] https://www.elastic.co/guide/en/elasticsearch/plugins/5.1/di...

If you are not using docker, you can manually add the specific privilege to open low-numbered ports without running the program in question as root.

We could expect every programmer to have domain knowledge of databases, even when it's not normally needed to operate one.

Or we could just make databases not acknowledge writes that haven't happened, not listen on all ports, not hide information about data loss in non-clustered setups deep inside the documentation.

Counterargument: many people who are good at a particular topic enjoy feeling superior to people who don't have their knowledge.

This is how this goes, and I've seen the pattern followed with several infrastructure services, even here in this HN thread right now:

1) A bug report comes in to AwesomeDB. "You do not have authentication support. This is bad. I'm going to yell about it on social media and make fun of you until you can accept a username and password, because everything in existence should write its own authentication code. Even MySQL takes passwords, dude, get with 2017."

2) A rough version is implemented by an intern, because authorization is way less sexy than hyperdimensional bloom filter quantum hash tables to speed up random reads. The intern then departs back to beer pong and nobody at AwesomeDB now understands the security infrastructure of their product.

3A) A mid-level executive, seeing the announcement, instructs me that I no longer have to keep AwesomeDB behind my 'stupid' reverse proxy + ACL solution and we can finally use native authentication and be really secure!

3B) Alternatively and more reastically, someone at Qualys adds "authentication is not turned on in AwesomeDB" to their scanning product and I get nastygrams from a contractor clicking "Scan" and copying and pasting the results for each of my fifty AwesomeDB instances.

4) During testing, I capture the credentials in plain text in about three minutes because it's a database. Why on earth would it terminate TLS? If we are down the 3B path above: Ha! Your move, security team!

5) Another bug report is filed against AwesomeDB to terminate TLS. Equal shaming is performed, because every service in the world needs to link in OpenSSL and terminate TLS, because that's never been a problem or backfired in any way. Oh yeah, and everyone has a different mechanism for handling certificates, and most don't bother to consider revocation.

(Just like the broader point I'm making about auth, stop adding TLS support to your systems. Please. If you speak TCP I can wrap you in something uniform that does a much better job anyway. You know how I know your integration is shitty? Because OpenSSL is hopelessly undocumented and the APIs even more so. You Googled and read a guide. How about instead, I can drop a pretty awesome TLS listener in front of you, potentially even in hardware! Whoo!)

6) TLS support is added to AwesomeDB. Cool! Now we can really be secure! (But nobody implements client certificates, so we still have to do the username/password shared secret business. And alternate DH parameters? What are those? They weren't covered in the guide, so AwesomeDB hardcoded a couple.) At this point, most people would put credentials directly in their repo. I'd sit around making the Vault argument, but nobody would listen. We have now traded my ACLs and authentication stack for security on our private GitHub repos. I set to work automating changing all of our infrastructure passwords every time an intern leaves the company. We still secure the network, of course, but now there's special strings of characters in existence to work around a competent network security strategy that already existed.

7) As a growing company we introduce LDAP and Kerberos. Oh wait, AwesomeDB and its home brewed authentication stack do not support LDAP because what startup needs LDAP?

8) Another bug report is filed against AwesomeDB for "LDAP support." A bunch of startup engineers spend several months trying to understand the guts of LDAP (I feel your pain) and, worse, the cornucopia of vendor shenanigans in that space, and set the bug report to "holy shit, someday." A rough version that works with one exact schema and hopelessly avoids being just right for my needs might ship, or might not.

9) I redeploy my perfectly functional reverse proxy solution which has supported LDAP since the mid 90s and wire it up to Google OAuth for bonus points (getting rid of passwords for all corporate systems overnight! Neat!) while half the world pointlessly fights for every infrastructure service that exists to basically link in an operating system's worth of encryption and authentication support just to be a document store or search server.

Given the choice I would very much rather secure my network or use well-established software to secure an infrastructure service. Asking every database vendor in existence to reimplement this stuff is a bad idea. People who blindly advocate for this sort of work, then paper over it with "you can't trust the network" without really supporting that statement, just generate busy work for companies that write infrastructure services. If you can't trust your network, what business do you have operating a production service that maintains user data? Do you really think having ElasticSearch do its own HTTP basic is more secure than any other solution? They at least had the JVM/netty/whatever to fall back on, where a lot of these infrastructure services have had to write their own authentication.

There is a middle ground here and a lot of advocacy, both on HN and elsewhere, that misses the forest for the trees. Of course, none of this applies to MongoDB, because they were like sure, the world needs another bespoke binary protocol. You absolutely should be keeping your data on a secured network, though, and opaque strings of characters for system A to log in to system B might seem like defense in depth, but they end up in your source code, another system, or the working set of your executing process, and they are routinely captured by active adversaries. So.

Having a mode where an infrastructure service assumes a connection is legitimate and just services the client is important. Some of us have a little more confidence in our security strategy than others and feel that we know what we are doing. Wiring that mode up to a 0.0.0.0 bind, by default, is the bug.

Oh yeah, almost forgot:

10) All that custom authentication in AwesomeDB that you had an intern write becomes a CVE factory and starts reflecting on the security of the product. Womp womp.

If you want to double check, try 'mysql --protocol=tcp'.

When you build a product or service, you should own it end-to-end OR simply let it be known that you are unable or unwilling to do so.

If you take it seriously, you'll know what can sink your project/business -- and stuff like leaving your entire database publically available with no auth, is one of them.