A Docker footgun led to a vandal deleting NewsBlur's MongoDB database
blog.newsblur.com
blog.newsblur.com
Should they have hardened in all the other ways for defense in depth, e.g requiring authentication from localhost? Sure. Should the attacker have not done the attack? Without a doubt. But should docker have not completely undone a security control?
Yes, and the fact that docker seems to have persisted with the current state is the topic of discussion.
Previously: https://news.ycombinator.com/item?id=27613217
I was running what I thought was an internal FTP instance for almost a week. Luckily it was about as hardened as regular ftp can be, but I noticed the problem when my service wasn't able to log in as the (very low) connection limit was filled by someone attempting passwords.
I've been using https://github.com/shinebayar-g/ufw-docker-automated to make docker compliant with ufw, and defining firewall rules as labels for the containers. It needs some work still, namely the service should be hosted in it's own container for easier updating, but it works reasonably well.
Totally unexpected outcome.
Luckily I discovered it in testing, so didn't make it to production. But annoying that those issues still remain for so long.
> This creates a firewall rule which maps a container port to a port on the Docker host to the outside world.
Source : https://docs.docker.com/config/containers/container-networki...
Requiring authentication from localhost does not seem relevant to me, given that the creds would be stored somewhere, either in memory either in a file anyway, but exposing a port is not "binding on localhost".
However testing your firewall after publishing a docker port seems common sense.
Indeed, that's how I found out Docker was using the DOCKER-USER iptables chain that you can customize:
https://docs.docker.com/network/iptables/
And that's how I made a simple firewall that works:
https://yourlabs.io/oss/yourlabs.docker/-/blob/master/tasks/...
Another thing, instead of using exposing ports like that, the easiest is to use Docker-Compose, so that your containers of a stack have their own private shared network, then you won't have to publish ports to make your services communicate. Otherwise, just create a private network yourself and containerize your stuff in it.
So for me that's two newbie mistakes which conducted to falling for this non targeted, script kiddie attack which has been going on since 2017.
But yeah, go ahead publish your ports instead of using docker networks how you should, "believe" in your firewall while you're at it, and then blame "docker footgun".
That only works with local communication unless you use docker swarm. So byebye high availability.
If you're going to make suggestions, at least think about them from a production point of view.
For example, I'm running a governmental, nationwide service handling 300k req/day, with 1k admins working daily on the site, on a single server without anyone complaining, and without less than 99.9% uptime even though I reboot to upgrade the kernel twice a month, I did have to fine tune the system but that was actually pretty easy.
In my personal experience, given a vendor SLA of 4:00 hours, Dell manages to send you replacement HW on time, but HP has up to now failed to deliver in window every time. YMMV of course. RAID is the only thing keeping us online in these cases.
There is also the normal server maintenance. Rebooting a kernel twice a month, if the reboot takes 2 minutes, already swallows 48 minutes.
Now to be honest 99.9% business hours only is mostly doable if you serve only 1 continent and manage to update/reboot outside business hours. I have a non-critical debian public web server running that updates and reboots every day at 3AM. Mostly because nginx didn't consistently apply lets encrypt certs. So far no one has noticed or at least complained.
Funny to see my above comment get a final score of -4, people don't like us saying "well just get a cheap linux box and we'll get you 99.9% uptime" ahah and this is "hacker news"!
Not to mention the comment that was like "you can't do production without HA", I have been developing, deploying, maintaining a bunch of productions for the last 20 years and very very few of them needed any kind of HA.
Anyway, with that experience I can get you >99.9% uptime with a 3 buck/month linux box, or HA for anything above 99.9, cause that's what experience is about kids ;)
Only on IPv4...
The machine with DB had a public interface. No matter firewalls, this is just bad. DB machine should be in private subnets, preferably with no inet access at all, even via NAT.
Proof that it had public interface. TFA: "Docker helpfully inserted an allow rule into iptables, opening up MongoDB to the world"
so yeah, it surely depends on the case given...
This is the type of thinking that lead to the attack in the first place. No, you should not pretend that your database is secure because it is hidden deep behind dozens of layers of firewalls.
You should assume that your database is always accessible from the interne and make sure that internet access to your database is not relevant.
With mongodb you are supposed to use TLS encryption and authentication even for communication within the same host because a low privilege hack or mistake could forward a database port or make it public.
In this scenario you not only need to bypass all the network security theater, you also need the password and a valid client certificate signed by a private CA.
It's clearly written in the docs:
To expose a container’s internal port, an operator can start the container with the -P or -p flag. The exposed port is accessible on the host and the ports are available to any client that can reach the host.
It should be common knowledge by now to either create a virtual interface / network on the host to publish ports on for usage by services that aren't running in Docker or (if you are in an environment where all services are in Docker) use --link between containers.
And as I understand it, this is very much an unintentional side-effect of ufw and Docker interacting – it's not Docker's the intention at all to override any iptable rules, just an unfortunate side-effect.
"It should be common knowledge by now" is very hand-wavy. I never used Docker much, and I could have been bitten by this. Where do I get this "common knowledge" from? Not everyone is a full-time sysadmin; some people just want to run a small service, read the basic documentation on various things etc., and set things up. Not everyone is super-invested in Docker (or any other tool for that matter) to have all the "common knowledge" that all the experts have. This is why defaults matter.
I get that Docker is often run by devs on their laptops, but this is bad behavior. The simplest solution I've seen so-far is Docker should only automatically set these firewall rules on devices that only have RFC1918 IPs. That would still keep things simple for devs on laptops, but would prevent firewall override on servers.
It is written explicitly in the manual that any port you choose to publish is reachable by all machines that can reach the host.
Just how more explicit does this warning have to be?!
And anyways: a simple "netstat -lnp" would have shown the docker-proxy process and world-wide reachability.
It's very easy to come along after something blows up, find something a person could have done differently, and blame that person, treating them as stupid or negligent. It's easy, satisfying, and often status-enhancing. It's also going to make the world less safe, because it prevents us from solving the actual problems.
Those serious about reducing failures should read Dekker's "Field Guide to Understanding 'Human Error'": https://www.amazon.com/Field-Guide-Understanding-Human-Error...
It comes out of the world of airplane accident investigations. But so much of it is applicable to software. The first chapter alone, the one contrasting old and new views, can be enough for a lot of people. It's available for free via "look inside this book" or the Kindle sample.
Admittedly, not as much (and since I've got a couple of years experience, back then the docs were bananaware), which is why I ran into the same issue - but one thing I always do is set up staging environments and check if at least the basic expectations (=can't reach an internal service from the Internet, but can reach public services) are working. Aka: a portscan.
A basic check would have prevented so many carelessly-configured services. Generally, it doesn't hurt anyone to run a nmap scan as part of the regular monitoring - if the line count doesn't match the previous one, raise an alarm.
My point is that this is the same behavior: looking for a reason why the person is wrong. That is a behavior that makes things worse because it takes the focus away from systemic improvements that make us safer. Please don't do it.
Systemic improvement would be to educate people to build fail-resilient systems. There are a number of reasonable precautions that could have prevented all of these "I exposed my unsecured MongoDB docker to the Internet" instances:
1) Dedicated hardware firewalls (or in the cloud, firewall rules): If all your server is accessible on is 80/443 anyway from the Internet, it doesn't matter if you punch a hole in the server's firewall.
2) Regular outside port scanning as part of your monitoring. This helps you catch misconfigurations of your firewall.
3) When planning a new infrastructure, think about which system needs access to which other systems. Your MongoDB doesn't need to have an Internet connection
4) Have someone experienced look over your staging system, do a full scale security audit or attempt basic measures of pentesting on your own.
5) Don't copy shit from stackoverflow or years old random blog post without understanding every single thing it does.
6) Don't use MongoDB which has been so fucking often in the news for being an insecure pile of dog poo that it physically hurts to read that name!
Organizations that fail to do at least the first four points as part of standard operating procedures deserve to be hacked, period. There is no excuse for slacking off on either of these in 2021.
Everyone wants to be treated as a (software) engineer here, but the engineer's perspective in this situation would be the opposite: The victims are the customers, and the perpetrator, acting in negligence, was Newsblur. Risk management is a core part of the engineer's job. https://www.sebokwiki.org/wiki/Risk_Management
There can be multiple victims, multiple causes/threat actors, and overlap between two categories.
But what's the end user a victim of in this case? Other than a brief lack of availability?
Of course the behavior of Docker is really problematic in this case, and ditto for MongoDB not having auth active per default. But both applications were consciously chosen by Newsblur.
I don't understand the relevance of this to my original question.
> Of course the behavior of Docker is really problematic in this case, and ditto for MongoDB not having auth active per default. But both applications were consciously chosen by Newsblur.
Or this, for that matter
---
So that you have it, my original ask was: But what's the end user a victim of in this case? Other than a brief lack of availability?
> It's not all that great blaming the victim.
Which is what I was replying to.
Of course this incident was inconsequential for Newsblur's customers. That in this case the impact for the customers was approx. zero doesn't change the relationship of Mongo+Docker <> Newsblur <> customers (i.e. who has responsibility and who feels the consequences, which defines who can be a victim), just the outcome for the customers.
See my previous note:
> There can be multiple victims, multiple causes/threat actors, and overlap between two categories.
I think we've both concluded now that newsblur was largely a victim as a consumer of docker.
Cheers, glad this helped.
Software culture is IMHO problematic there in multiple dimensions.
No, the parent is a victim and the children are dependents of the victim. Thus hurting the parent also hurts the children. It's the same thing with Newsblur. Hurting Newsblur hurts their customers.
* Run a SaaS that costs money
* Do not perform the migration in prod replica setup with test data and monitor for, observe and fix any oddities
* Cause a 100% data breach by not following a sound change management procedure
Just because this individual is being blamed and not a Ltd. or Corp. doesn't mean that this is how a quality SaaS should be run. I know that more it's-just-a-side-gig SaaS wing it more than they do not but this is exactly what happens when you haven't got good ops. Rather than random error, this is a consequence from a systematic error.
Of course, as seen here, a network service binding to more than localhost is strictly more exposed - and auth should be required in those cases (even when the service should ostensibly be LAN-only).
Secure-by-default is the best kind of security.
If you’re writing software that deals with network connections (as a client or server), write the code for UNIX sockets first. It’s usually trivial to bolt network connection code on top, and being able to deal with sockets directly can make automated testing significantly easier.
It has an added benefit of not needing vendor-specific TLS configuration and firewall rules easy to configure since everything goes through the single VPN port.
'easy to start developing against' falls flat on its face once you deploy to production and realize you have to bolt on authn and authz to your code and data connections instead of properly designing it in from the get go. (it might work if you never deploy your products though. god knows how many products don't see the light of day.)
I would blame MongoDB for its default-insecure configuration. There's no excuse. It's been like that for at least a decade (when I last used it) and it was a bad choice even then. At a _minimum_ when they upgraded the engine to integrate WiredTiger they should have pushed that through as part of the breaking change.
What system, framework or stack are you using that stores the code of the job in the database? I'm curious, because I can imagine it solves some issues, e.g. deploying new code while allowing old code to finish running and scheduled jobs.
Also, if you have stuff like this in Redis, which will get picked up by a job runner:
{"runjob": "foojob", "params": ["one", "two"]}
- run jobs that aren’t supposed to run (removing an account for example, is that’s scheduled after a 2 week grace period) maybe an export or import job. Can be anything of course
- if your job runner allows scripts, or arbitrary class methods, you can do whatever you want
- you can remove jobs if you feel like it
- if you can escape redis because of whatever exploit, you now have access to the internal network
In general I use sidekiq, but resque and inspired implementations generally work on simply calling perform(), so any class with such a method can be called, depending on the typesystem.
The biggest two issues are the ability to perform any defined task, and the larger attack surface of an exposed redis server
And the hacker can still delete your database or steal your data. He will simply download the datafiles and then delete those data from the server. Drop database done without a database password. And no, databases aren't kept encrypted on the server.
So let’s start blaming every piece of software that listens to localhost, but is exposed by docker.
Not a fan of mongo, but the defaults are way better than before.
Docker needs to fix their defaults
Yes, root account password and access permissions should be changed upon a fresh install, but the real issue here is Docker's "helpfulness" by opening ports without explicit permission. That's absurd, and has no reasonable excuse.
> Installation of MySQL creates only a 'root'@'localhost' superuser account that has all privileges and can do anything.
It's only a local account. Sure, it would be great to have a password on it, but it's not "a completely open root account".
It has a password, which the user has to change, before they can do anything else. From that same page you linked:
""" For data directory initialization performed manually using mysqld --initialize, mysqld generates an initial random password, marks it expired, and writes it to the server error log. """
This is a feature and a dangerous one.
Our provisioning had to carefully manage the firewall during installation, because MySQL would be open to the world *while provisioning*. Often this is mere seconds or less, but on a reasonable popular webhosting, this is enough time for automated tools to take over the server. If, during that provisioning some other tool would start punching holes in the firewall you carefully manage from provisining, I would be enraged. I dislike Docker for a lot of reasons[0], and this kind of behaviour is one of them.
So, yes: what MySQL or MongoDB do is dangerous and poor security practice. But what Docker does is worse: it breaks the countermeasures you take and rely on. IF all your layers of protection are removed by poor security practices of a tool that says "yea, but you need more than just this one layer" you still have no protection.
[0] yet I do use it a lot on development-machines to run the services needed on localhost and not have to manage all those with apt/snap/brew or whatever they come with.
That's flat out wrong. You can't start MySQL/MariaDB docker images without either explicitly specifying a root password, have it generate a random one on the first start of the container, or explicitly allowing an empty password.
Regarding native installs of MySQL/MariaDB, the situation is a bit more murky, but at least the Ubuntu/Debian packages will ask you for a password during setup (and the bind-ip is 127.0.0.1 by default which means you'll have to mess around with conf.d files if you want to setup an externally-reachable server).
It's Docker's responsibility to not open firewall ports unless explicitly asked to do so.
Opening random ports without A) Telling the developer first, B) Requesting permission to do so, C) Explaining why Docker wants to do so and D) Providing the ability to "opt-out" of this insecure-by-default configuration - is very bad.
It doesn't matter how secure or insecure the software inside the container is... it shouldn't be allowed to communicate with the public unless the developer specifically requests it.
You are asking Docker to open firewall ports with the -p option, it explicitly even says in the manual that published ports will be reachable from everywhere that can access the IP you're publishing on (either default 0.0.0.0 or an IP of your choosing).
I don't have a horse in this race, as I don't use Docker for a plethora of reasons dating back to it's inception. There are plenty of far superior containers to use. Just adding this footgun to the list of reasons...
Regardless, a supposed "enterprise" tool shouldn't start changing your config in surprising ways, like punching firewall holes without explicitly being asked to do so. That's just crazy...
The packaging does the necessary setup for you, and modern MySQL has secure defaults anyway.
They had super lax security, making them easy to use for newbie devs, who are frequently scared/easily distracted by security settings.
I'm quite convinced the lack of security was by design. Growth hacking and all that. Get everyone onboard and once you have big business going on, you can focus on the minutiae of security, scaling, not losing data.
Node.js, PHP, Docker (heh!) most other popular techs did the same thing: super lax and frequently wrong defaults, just to get adoption.
As a startup it seems you either choose right/security or wrong/money :-)
The development strategy did work, though.
right/no users or wrong/yes users
Not much point being right if no one uses the product :)
My point was: why didn't the author of the article enable the use of authorisation? To rely on only a firewall is just stupid.
Looks like the author is ahead of both us. Later in the post he mentions both a planned transition to a VPC, and plans to beef up the security of MongoDB itself.
Docker altering firewall rules without explicit instructions to do so is either a feature, if you're looking to defend docker, Or a security defect if you're feeling rational. There's an argument to be made for the root cause of the issue, The docker container manifest, or the application itself. But to pretend like this is what docker should be doing, users should know better just seems wrong.
The funny thing is there are definitions for it in wikionary and urbandictionary yet it is picking the HN one.
There exists an entire class of software that is best avoided. Looks like their usage is highly correlated. As a parallel, I was just saying in a work meeting that the first step towards a good IT environment is not adding some crazy non-causal component like our F5 middlewhare, while waiting for a command to end on our Oracle database. That other class is highly correlated too.
1. Docker doing this iptables change out of the box, and
2. MongoDB not having a password set out of the box
The life of a developer (and solo dev) means you often have limited time you need to navigate a project and try and do your best to understand, deploy and use it -- this is just one of many tasks on your TODO today to get you closer to operating your product.
I really wish these kinds of things were more secure in a few ways:
1. Defensive defaults (passwords, not opening holes in firewalls), and
2. Not making the security hard to use
If its painful to work with the security feature and "get it working", someone with limited time may just undo the defensive defaults to get things working again (doh).
But I get it, sometimes a security piece on by default is so cryptically painful to understand that you get so frustrated with it you just turn it off.
I think what's missing is an easy to use tool, installed by default - you run it and see a clear overview of your system security - what ports are opened, how is SSH accessible, how secure are the running services, etc. with plain ticks/crosses to show what's good or not. The kind of tool that a developer can install on their server and then check that at least the basics are right.
> it seems to give a lot of engineers into thinking they know infra when they really don't
Maybe, but technology changes over time - I don’t see many new projects choosing VMware over docker/OCI for new infrastructure deployment since you usually don’t need a full VM for applocations that just need isolation and easy static deployments.
There's a whole generation of sysadmins that use docker so that they can stay away from foundational knowledge. We interview experienced devops who do not know/understand how to build basic packages from source (e.g. they don't understand the ./configure, make, make install chain) and who only have basic knowledge of the underlying operating system.
But so far, life has been too short for me to waste my time on Gnu Autoconf. And I don't feel guilty about this, or like someone who doesn't know fundamentals.
Autoconf is by all accounts a horrible system. Gnu Make ain't much better.
So I can't fault people for trying to avoid this mess.
Autoconf is just really terrible.
Sendmail is another great example of this phenomenon.
Just like PHP and COBOL solved real problems back in their heyday.
They are just not something anyone should be using nowadays.
(Btw, Facebook's Hack is a surprisingly pleasant language. Probably the best one they could have made, coming from PHP.)
I guess it depends, take Vim for example: been around since the early 90s, the developers are familiar with autoconf, and it seems to work well for them. Why fix something if it ain't broke?
Switching build systems is not necessarily an easy task. Also, as a user I always found cmake quite hard to deal with. Actually, as a user, I actually prefer autotools in spite of its shortcomings. I think there's still quite a bit to win here.
Either way, I personally wouldn't really call autoconf "terrible".
And, yes, switching an existing system is a different proposition from starting from scratch.
Though see what the Neovim people have been doing to remove some of the accumulated cruft from Vim, if you are interested.
Ex-Amazon here. Being deeply familiar with OS fundamentals and internals, "low level" tools, and so on is crucial.
A lot of candidates show to interviews with CVs filled with names of popular frameworks and fancy devops tools and often don't understand what really happens behind the curtain.
People are becoming less familiar with the basics and tend to reinvent the wheel. In many teams this does not get you hired.
> Being deeply familiar with OS fundamentals and internals, "low level" tools, and so on is crucial.
...needs to be front and center on the job description then. Far too often these kinds of things are hidden on the initial job descriptions and it just wastes everyone's time. I get that HR/recruiters want to throw a big net but vague job descriptions are often part of the problem.
docker breaches the localhost VS external port security model (and others)
I don’t say this lightly: this is a big problem in our industry right now. DevOps means (to some) that developers now handle operations. The reality is that it’s difficult to juggle an operations mindset with a feature driven one.
Even if you focus on infra problems full time there is so much ground left uncovered, it would be impossible for anyone to juggle all of dev and all of infra at once, and this is compounded by the fact that ops is reductionist and dev is additive, which is an incredibly difficult problem for a person to reconcile and give full attention to one side. That’s why devops was supposed to be a division of labour; not just a dude/dudette who can configure nginx and write code.
Reminds me of this: https://www.weforum.org/agenda/2021/04/brains-prefer-adding-...
Side note: everything that’s ever existed is “flawed,” it’s a weird word to use in the context of something you want to discredit, because then your alternative had better be “flawless” and it obviously isn’t.
...yet MongoDB's (default) configuration assumes there is. That's the big problem with perimeter security: applications offloading their security responsibility to other, possibly imaginary, parts of the system.
Docker diligently sidestepped that firewall, and in so doing exposed that this was a case of perimeter security. Because by bypassing that single external filter, the entire service was now vulnerable.
Obviously with only perimeter security, those servers are soft targets to an attacker who compromises a frontend host. I am all for hardening the interior.
"Don't put stuff on the internet that doesn't need to be, even if you think it's secure, because it's probably complicated enough for you to be mistaken about that." This is a perimeter security philosophy, and also what OP needed. If anything the host-level firewall mishap seems closer to an application-level authz bypass than to a pivot across a "trustworthy" network.
This feels like an odd statement. Surely a database being dropped all at once is about the loudest possible thing that could happen to a database-reliant application?
It'll be slow at first, and you'll need new rules each time you bring a container up but you'll have far more control.
We switched to this approach when we had an IP dual-stack issue with the proxy and not only do we know it's secure because we wrote the rules, we get better performance without the userland proxy and more control over routing.
It's important to understand that because your containers sit on a bridged network, INPUT rules do nothing to stop incoming connections, it's all about the FORWARD table.
Edit: apparently there is a sysctl setting that makes the bridge netfilter code call the iptables filter routines (bridge-nf-call-iptables) so i am kinda wrong...
Having root in a VM doesn't typically give you any rights on the hypervisor (at least not on eg Xen).
On a hypervisor, it's much harder for VMs to influence each other.
Linux containers (and docker amongst them) started out as convenient and reasonably performant, and added security later. One patch at a time.
Historically, hypervisors typically started secure and added performance and convenience over time.
(Very simplified. But I used to work for XenSource back in the day.)
https://www.linode.com/blog/linode/cloud-firewall-beta-open/
You have to tell fail2ban to use the DOCKER-USER chain when adding rules, otherwise none of the rules it adds do anything for docker.
(Unless you know to use -p 127.0.0.1:1234:1234 instead of just -p 1234:1234 like all the examples on the web tell you.)
The OP didn't (I think) say but... were they running their containers with the --expose or --publish options? Without those, there shouldn't have been anything listening. And with them, well, the clue's in the option name...
That said, I've never liked the way docker implements its networking. Not only does it monkey with iptables (not too bad once you expect it) but, worse, it silently enables IP forwarding on all interfaces! Any system where it is installed is now a router; this should make anyone who uses a VPN to connect to work rather nervous...
I really wish docker used a different solution for exposing container ports by default, such as having the docker daemon proxy traffic. Yes, it wouldn't be as fast. But this wouldn't matter for the vast majority of users, who would benefit from having behaviour that matches other listening ports, doesn't bypass the firewall, etc. Have the iptables implementation available as an option that is there when you really need the performance--and know what you're doing!
However, the much worse problem seems to be the fact that NewsBlur didn't test their network connectivity with something as basic as a port scan. This wasn't some complex attack based on some complex code injection jumping through legitimate ports or anything: their machine had the Mongo port open when they thought it should be closed. This is the kind of thing that absolutely shouldn't make it past basic testing.
You don't write a program and deploy it without testing, so don't do the same with your firewalls. You need to actually test your firewall policy and not blindly assume it does what you think it should.
Making mistakes like this sucks but passing the blame makes it even worse.
As a Newsblur customer, would have really preferred the following course of action: lock down infrastructure, confirmation that no leak occurred, communication of that fact to customers and THEN service recovery. While this was all happening, the only indication that most likely no data was leaked was a single HN commenter that had a similar experience: https://news.ycombinator.com/item?id=27615708
Regardless of the docker foot gun this could have been a human fudge as well.
Having done exactly this myself before with SQL server and Slammer I didn’t blame the default configuration, I blamed myself.
Docker on Ubuntu has a critical security vulnerability, with a track record of being exploited. That, clearly, is something that should be fixed ASAP.
If you have just a single layer, sure, blame yourself for poor security practices. Services should listen to unix domain sockets or internal networks only. They should have authentication in place. Machines should not have routable IP addresses. Machines should have externally applied firewalls. And so on. That doesn't mean the security layer that did fail shouldn't be fixed.
Even temporary rules and default states aren’t protected against until your automation is complete if your box is on the public internet. Or if you screw your automation up, the same happens.
I really hope they learn from all this the right lesson, which is not to just blame docker, but to carefully think whether you really need to use the Internet as a mean of internal communication for your services.
You don't need to map ports for inter-container communications, and maybe that's where some of the confusion comes in for people using docker.
I have noticed a few friends who have been playing with docker for home server stuff thinking that they need to map ports for everything, just to get containers in a stack talking to each other.
At the same time docker really shouldn't be messing with the hosts incoming firewall rules.
And MongoDB really should force auth by default.
> I switched the MongoDB cluster over to the new servers.
It sounds like they had multiple Mongo nodes on separate machines, communicating over the internet. Presumably the application(s) are also on separate machines and would need to communicate with Mongo.
I think `docker network` has options to create a network across Docker daemon hosts? But that has some tradeoffs, I'm sure.
Block the ports using private networks or use the IP security provided by the cloud provider (which does not use your machine's security to accomplish this).
Defense in depth and all that.
But sure, docker has a footgun, everything does, otherwise linux would not ship with sudo. (someone's going to reply with a linux that does not come with sudo).
To be fair, it did not look to me that docker intentionally punched holes in firewall- it purposefully places it's rules in separate table/queue, which due to way iptables work, hampers the firewall rules.
However, once it became apparent to docker developers, it should have been made a priority to be fixed, as it is counterintuitive default.
“sudo ufw status verbose”
I expect to see open ports in my system.
It’s an issue that ufw doesn’t show some open ports (all relevant iptable rules).
docker run -p 127.0.0.1:5555:5555 ....
I never really liked the whole container idea, but this is ridiculous.
1. `docker run -p 0.0.0.0:6666:6666 ...` - docker will update the firewall to allow traffic to port 6666
2. `docker run -p 127.0.0.1:6666:6666 ...` - docker won't update the firewall.
3. `docker run -p 6666` - the port is published on 0.0.0.0 and a random port is opened on the host
I see 3 being a little surprising, but 1 and 2 does what I would expect of it.
I don't think Docker should touch the firewall whatsoever. Apache doesn't, nginx doesn't, PostgreSQL doesn't, etc. Why is Docker different? The sysadmin should decide that!
`docker run -p 6666:6666 ...` which defaults to `-p 0.0.0.0:6666:6666`
I can't find it in the article, but I suspect this what actually happened. It's a bad default and unfortunately most examples build on it.
Similar, it's easy to deploy MongoDB without a password. But it's also easy to set one, the docs even mention it https://hub.docker.com/_/mongo
And everyone can run a portscan with nmap against their servers. Unfortunately the MongoDB default ports 27017-27019 are not part of the top 1000 ports which nmap scans by default.
I remember experimenting with docker running on a VM, and being confused for a long time why my ufw rules didn't seem to be working, until I found out this was a known issue.
These days its better to use the hosting provider firewall on top of ufw. Leading providers like AWS, Hetzner etc all provide the feature to add firewall rules via their UI.
General advice - perform threat modeling of your services to uncover weak links.
Can Docker really be blamed here?
It sounds like when they ran MongoDB they explicitly published Mongo's port to the internet by either adding the ports property to docker-compose.yml or the -p flag with a Docker command to open Mongo's port to the outside world.
Not to pour salt on an open wound but why are they blaming Docker for this? Surely an ops person should have thought to check if any internal services are exposed to the outside world? This goes with or without using Docker. This would be like explicitly opening a port with iptables by copy / pasting something off the internet and then blaming iptables because it opened the port.
If anyone is curious, years ago I've blogged about the difference between expose and publish for Docker at https://nickjanetakis.com/blog/docker-tip-59-difference-betw....
This whole incident would've been prevented and hopefully eventually rectified if they'd put auth on the localhost interface as well.
Let your tools save passwords, but it's insanity to have anything just blindly accept requests - you're one mis-directed script away from people wiping out your prod DB, let alone dedicated hackers.
I remember back in 2016 I had a similar issue with Redis. I ran it with -p 6379:6379 with no password and someone got into my Redis container back when "crackit"[0] was a thing.
But I never blamed Docker or Redis. I blamed myself because I was careless for not fully understanding the implications of what -p did at the time.
That said, it also seems like the docker images are a different beast, so be careful what containers you ship and with what configuration…
Does anybody know if this threat is at all credible?
Relatedly, you can be fined for not telling the regulatory authority about breaches yourself: "In the case of a personal data breach, the controller shall without undue delay and, where feasible, not later than 72 hours after having become aware of it, notify the personal data breach to the supervisory authority competent in accordance with Article 55, unless the personal data breach is unlikely to result in a risk to the rights and freedoms of natural persons."[2]
British Airways[3] for example was fined for inadequate security measures. I also remember a case where some company (cannot quite remember which) itself told the regulator about some security snafu as they should, a snafu that most likely was not exploited. They were still issued a fine but a reduced one because they were very cooperative and most likely no actual damage happened.
There is an enforcement tracker website[4] where you can filter for e.g. "security" in the type column.
But nobody will go to prison for being the victim of a hack, that's just fearmongering by that attacker.
[1] https://gdpr-info.eu/art-32-gdpr/
[2] https://gdpr-info.eu/art-33-gdpr/
[3] https://ico.org.uk/about-the-ico/news-and-events/news-and-bl...
The clear mistake here is allowing incoming traffic on 0.0.0.0/32
Disclaimer: I am not into Docker or Mongodb
It's more about the setting of the ufw. Docker relies on their own NAT table, and setting ufw accordingly requires a bit of setup. That said, having a good setting from hosting provider is quite easier ans safer.
Docker is 100% to blame here and they need to change the default! Just put it in the release notes and get it over with.
I agree with the other posters, that Docker can cause problems with firewalls, as pointed out both in the article, in the GitHub issue, and in the other thread as well: https://github.com/moby/moby/issues/4737
Furthermore, it also seems to me, that not only Docker should respect the firewall rules (even though it'd confuse another group of people about why their services aren't accessible externally and would necessitate manual firewall rule management, short of some explicit way to do that, such as docker run ... -p 80:3000 --expose-in-firewall 80 ...) and that MongoDB also should have secure defaults, as any other piece of software!
That said, at least to me it appears that Docker Compose, Docker Swarm and other technologies have attempted to introduce a networking abstraction to allow running services (more) securely and privately, by not exposing their ports on the host directly.
For example, see the following example of a Compose file, which would only expose a web server to the world:
version: '3.4'
services:
# you would probably want to run MongoDB not exposed to the outside world at all
# notice that there are no ports exposed here, given that we're using Docker networking
mongodb_container:
image: mongo:bionic
restart: "unless-stopped"
environment:
- MONGO_INITDB_ROOT_USERNAME=lazy_example_of_a_single_file_config
- MONGO_INITDB_ROOT_PASSWORD=but_you_should_be_able_to_store_this_in_secrets
# see docs for MONGO_INITDB_ROOT_PASSWORD_FILE at https://hub.docker.com/_/mongo
deploy:
replicas: 1
placement:
constraints:
- node.hostname == your-mongo-server.com # just an example, probably use labels instead
resources:
limits:
cpus: "4"
memory: "8G"
# your web application wouldn't need to be exposed to the outside world either
# it could just receive the requests from a web server that ensures SSL/TLS
web_application:
image: hello-world # which you would replace with your actual app
restart: "unless-stopped"
environment:
- APP_CONFIG=variables_go_here_or_in_a_config_map
- MONGO_HOST=mongodb_container
- MONGO_PORT=27017
- MONGO_USERNAME=either_the_username_above_or_a_non_root_one
- MONGO_PASSWORD=though_the_non_root_needs_separate_creation
# maybe with a helper container, like some people use dbmate for RDBMS
deploy:
replicas: 4
placement:
constraints:
- node.hostname == your-application-server.com # just an example, probably use labels instead
resources:
limits:
cpus: "2"
memory: "2G"
# finally, the actual web server is the ingress point, which you can self host
# in the case of Kubernetes and other fancy tech you'd be able to use external solutions, probably
web_server:
image: caddy:alpine # or another web server, that would actually be exposed
restart: "unless-stopped"
ports:
- 80:80
- 443:443
# though sometimes you need the long format for the above so client IPs resolve correctly
volumes:
- some_config_directory:/etc/caddy/Caddyfile
- some_other_config_directory:/data
- yet_another_config_directory:/config
deploy:
replicas: 4
placement:
constraints:
- node.hostname == your-ingress-server.com # just an example, probably use labels instead
resources:
limits:
cpus: "2"
memory: "2G"
Docker even has a page on networking, that provides information about the additional functionality that's available as well: https://docs.docker.com/network/
Of course, that's not to say that defense in depth and external solutions wouldn't perhaps be a better option.
why the redaction? name and shame.
The latter is a mistake. It doesn't ever make sense to expose hosted container ports to host, with the exception of public-facing stuff (like a webserver). The whole backoffice should be on a separate network.
And this is how Docker works by default, since forever, unless one very explicitly requests that it exposes ("publishes") the ports.
Just don't expose stuff. If the host needs to talk to a database or something, it can always talk to it via that `docker0` (or however it's called, I don't really remember the details) interface. (I believe it doesn't always add a route, because on one of my machines I have to manually add a route so Traefik on a host is able to talk to the containers on a isolated virtual network.)
Yes they should’ve configured, checked, etc. But defaults sound be secure by default
Relying solely on a host-based firewall for access control is, for reasons which must now be obvious, admin incompetence.
They are responsible for securing the containers. They didn't.
Your services should be using authentication even if they are only bound to localhost.
If you're backing up diligently, have a problem, go to restore backup, and discover that the backups are unusable due to an issue caused by the backup software's default configuration, you could say "the mistake here is you relying on a single backup system, pardner, not the backup software's bad default configuration," and in the sense that it's good advice to have multiple backup strategies, sure, you're technically correct. But that doesn't excuse the backup software from having a dangerously bad default configuration. That would be relatively trivial to fix. That they've apparently known about for years.
The blame game doesn't help solve the real problem that people are unknowingly putting services directly on the public internet when they think they are secured by a firewall. With how many millions of people use docker every day IMHO the severity of this issue is far too great to ignore. A simple mitigation to fail and warn a user when it appears they might be hitting this issue (i.e. before docker run changes iptables config, give a quick look to see if it looks like it's already configured for the most common ufw or other setup) would stop this from happening for 99% of cases.
Everyone knows docker runs as root, and therefore be careful using it, just as you would sudo. It seems this is a major fail in docs.
I kinda get why it's the default, but it needs to be made way better known that by default -p punches a hole through ufw.
You're confusing the proximate cause for the root cause.
Unauthenticated mongodb instances are a pretty common problem - it's why a "script kiddie" was so successful.
Security specialist here. Startups are built by generalists. Good decisions were made here which made other defense-in-depth considerations not as critical when reconciled with go-to-market needs.
If every generalist focused on every security risk in their product, they'd expose themselves to the business risk of not moving quickly enough.
It's pretty clear NewsBlur did what they could and relied on the expertise of others to not fail them in exceptionally basic, entirely avoidable ways. They were betrayed by that reliance.
The core message ("using auth on MongoDB would have prevented this, it's always a good idea to add password auth just in case") is perfectly reasonable; we can all learn from this, and it's perfectly fine to point out such things.
But the way it was phrased was absolutely not okay. People make mistakes all the time and they are not "incompetent". This is the classic "I am very smart, I never make mistakes, if you made a mistake then you're an idiot. You probably eat poop. I am smart btw"-attitude that's just ... ugh...
People rely on firewalls to prevent mistakes from becoming disastrous. Defeating that silently is super surprising. People don't know everything about every piece of tech they use either; very few people do: it's just too much information.
And it's not like auth alone is perfect. Remember when a bug in MySQL allowed people to bypass the auth? Good thing I put a firewall in front of my Doc... oh, no, wait...
So this is why I downvoted it.
It's not an insult to call someone incompetent.
Yes it is, especially when based on a single data point.
It can be both.
It’s a docker problem, which should listen to localhost by default
If you read the Footgun at https://github.com/moby/moby/issues/4737, this is exactly what happens: someone sets up a conservative firewall, then Docker drills holes in it and opens itself up to the world, regardless of your firewall.
Yet, what Docker does is still wrong: it should not disable Ubuntu's default security. Ever. Even if that security is inadequate. An analogy would be to say "While watering your plants last night, I left all your windows open. Now your stuff is stolen. I did this because you should have installed window grills: without window grills your security sucks anyway."
This, the backups, the write-up, all make it really hard for me to want to victim-blame the dev for not catching a very silly Docker default.
That having been said - public ip? - no fw? - no password on the mongod instance?
Idk, couldn't be me, not even in dev, just take the 20 seconds to plumb the pw to both sides. Modding me down won't change these facts and won't keep you from being compromised if you take the same lazy steps
That said, some external firewall would prevent this. My VPS provider allows me to configure which ports I want to expose to the outside world which would mitigate this kind of issue.
If I run a docker container on my macbook and expose a port over 0.0.0.0 (docker run -d -p 8080:8080 nginx) and it's available on my LAN network, how does it get exposed over the internet? Unless there's a rule in my router/gateway which does the port forwarding, it's not possible.
I wish the author goes into a bit more detail onto how his setup was vulnerable rather than bringing up an old Github Issue.
On your system docker is running in a little virtual machine that has its own linux kernel, virtual peripherals, etc. This gives you an extra layer of security (by pure happenstance) such that the VM has to allow traffic in too.
And this is kind of why this issue is so nasty. You as a mac user might never know that this is a problem or potential footgun. You could happily develop entire production services on your mac, then move them to a shared linux host like Digital Ocean and... uh oh, now your services are open and you had no idea it was even possible.
If you're deploying code without knowing how networking works, you're always going to have problems with this like this.
Perhaps my example was a little too simplified as AWS is my day-to-day cloud host but I find it hard to believe opening up ports on your VM and exposing it to the world and then placing the blame on docker is fair.
It's also fairly normal for a server to have public IPs associated with it directly, and for those IPs to be available to the public internet directly.
The latter is what probably happened here.
Newsblur appears to run on digital ocean, where I believe they still default to associating a public IPv4 address to each droplet, which again is pretty normal in the server world.
So there was no firewall then? https://www.digitalocean.com/blog/cloud-firewalls-secure-dro...
So if you’re not careful you may be exposed on ipv6 even if incidentally protected on ipv4 by the NAT.