A Docker footgun led to a vandal deleting NewsBlur's MongoDB database (2021)

Thread from previous discussion https://news.ycombinator.com/item?id=27613217 (June 4, 2021)

It's a pity that the chances of nailing the perps is so low. Obviously docker and the person that put this together share some of the blame but: the original internet would have never gotten off the ground if it wasn't for people cooperating with each other rather than to try to tear things down all the time.

And with the chances of your average script kiddie/hacker/idiot getting caught being lower than a typical bike thief in Amsterdam this likely will not stop. It would be great if we could get a secure internet. But what we get instead is an internet that can only be kept running by large companies because they supposedly have the budget and the knowledge to do this at a scale that makes security affordable. Which means that a ton of innovation will simply never happen because for a large number of fledgling companies the decision between working on their business or working on the security of their business is a moot one, if they don't do the one the other will kill it and doing both at the same time is too costly.

NewsBlur is useful, destroying it serves no purpose at all. And make no mistake: the hacker clearly aimed to simply destroy it and pretend they have copied the data, so they were more than willing to do just that, wanton destruction for a miserly amount of money. Whoever did this may think they're l33t and cool but I personally think they are utter trash.

Am I missing something? The article wrote:

> When I containerized MongoDB, Docker helpfully inserted an allow rule into iptables, opening up MongoDB to the world

But the blog post doesn't mention how Docker "helpfully inserted an allow rule". Is this because NewsBlur ran the container using the -p 27017:27017 flag without reading the docs around what publishing a port does?

You don't need to publish a port for (2) containers to talk to each other, they can do that over a private Docker network that both containers belong to (something Docker Compose does for you by default). You can also choose to -p 127.0.0.1:27017:27017 which will only publish the port so that only localhost can access it, something like this is handy if you publish a web port to localhost so nginx not running in Docker can connect to it.

"When I containerized MongoDB, Docker helpfully inserted an allow rule into iptables, opening up MongoDB to the world."

Yet another reminder that the most important ability in systems engineering is good judgement.

Fantastic write-up.

This seems to me like a combination of multiple foot-guns, first being the Docker one - followed by the fact Mongo was not configured to authenticate the connection.

Heroku by default run PostgreSQL open to the world (which is problematic for other reasons) but they get away with it by relying on PG's decent authentication.

My default is to prefer to build systems with multiple layers of security, such that there is no reliance on a single issue like this.

Great write-up. I remember how shocked I was a few years back when we moved our backend to a micro-service architecture and our native Postgres installation to a docker container. As (I suppose) almost everyone, we also used ufw to manage the firewall.

Security should be the default stance, and any port exposed through docker should initially be restricted to local only. Going global should be explicit, imo.

I am not able to follow the reasoning of this part of the write-up:

The most important bit of information the above chart shows us is what a full database transfer looks like in terms of bandwidth. From 6p to 9:30p, the amount of data was the expected amount from a working primary server with multiple secondaries syncing to it. At 3a, you’ll see an enormous amount of data transfered.

This tells us that the hacker was an automated digital vandal rather than a concerted hacking attempt. And if we were to pay the ransom, it wouldn’t do anything because the vandals don’t have the data and have nothing to release.

Personally I blame the irritation and cost of setting up private subnets that can access the Internet via NAT in AWS VPCs in addition to Docker's interactions with UFW. Private VPC subnets with NAT Gateways are both unacceptably costly and complicated to setup (source: recently dealt with the combination of NAT Gateways + the Network Firewall service in AWS). Throw in the need for VPC endpoints for many workloads and AWS networking starts to look like an extortion racket for those who want the majority of their infrastructure to be sequestered from the Internet.

Great writeup! Writeups like these help everyone learn from each other's mistakes.

> We had been relying on the firewall to provide protection against threats, but when the firewall silently failed, we were left exposed.

> a change needs to be made as to which database users have permission to drop the database

I think these two definitely highlight the importance of the always using the "layered onion" model of security.

The combination of a firewall, a password, and a nuanced permissions model would have been sufficient to mitigate the attack, even if one or both of the others failed simultaneously.

Just wanted to say that I recently started reading RSS again and NewsBlur has been excellent. It’s a good aggregator, has a great web interface, and a great iOS app. The free tier is all you need (so I guess I’ll need to pay just out of my desire to support).

Nice read. It might be a good idea to also lock a user account after N failed password attempts. Mongo does not seem to support that off the shelf - https://www.mongodb.com/community/forums/t/limit-failed-logi...

Neither do other databases like PG, curiously enough. The recommendation seems to be to link to LDAP or use authentication hooks.

Or perhaps use client and server certificates for increased security - https://www.postgresql.org/docs/current/ssl-tcp.html#SSL-SER...

Oh that's funny, we were just talking about docker and ufw mishaps over here: https://news.ycombinator.com/item?id=34858775

> Turns out the ufw firewall I enabled and diligently kept on a strict allowlist with only my internal servers didn’t work on a new server because of Docker.

OOF. Those can be nasty. Many of the tools "helpfully" inject jumps to their chains at the very front of the table, libvirt example:

    Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
     pkts bytes target     prot opt in     out     source               destination         
    9105K 4151M LIBVIRT_INP  all  --  *      *       0.0.0.0/0            0.0.0.0/0        

    Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
     pkts bytes target     prot opt in     out     source               destination         
        0     0 LIBVIRT_FWX  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
        0     0 LIBVIRT_FWI  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
        0     0 LIBVIRT_FWO  all  --  *      *       0.0.0.0/0            0.0.0.0/0

    Chain OUTPUT (policy ACCEPT 229K packets, 39M bytes)
     pkts bytes target     prot opt in     out     source               destination         
    6893K 1628M LIBVIRT_OUT  all  --  *      *       0.0.0.0/0            0.0.0.0/0   

So if you relied on your firewall blocking access and didn't test it it's easy to be bamboozled.

The common approach is to allow other tools only manage their own chains and call those chains from your <firewall tool> but not that many tools allow for that and want to take over your firewall.

>Before the former primary server could be placed into rotation, a snapshot of the server was made to ensure the backup would not delete itself upon reconnection

I don’t understand this. Was the snapshot made of the compromised database or of the old not in use one? Why would the backup (snapshot?) delete itself on reconnection?

Looking at their self-reported realtime stats makes me wonder why this architecture. The current numbers imply peak database request rates in the hundreds/second.

This appears to me like another case where we could have built a simple, 1-process vertical on a single box and called it a day.

This happened to me as well. Only that it was on a toy server for a side-project.

https://twitter.com/joaoqalves/status/1332638533572550656?s=...

db on public subnets are gross.

if it’s unavoidable, then monitoring your db isn’t reachable from the public internet is important.

Kinda weird to blame Docker. This is pure user error.

I'm glad this wasn't another "database lost, we have no backup" stories.

Lol, that’s the same footgun I discovered myself when was checking open ports. Who that wise guy in Docket team who decided to pass default firewall rules and open containers ports to public?

You should consider yourself lucky that you didn't lose your users data.

My 2c:

- NEVER expose a database to the public. Use at least a micro-service BFEs that do exactly what the app needs and nothing more.

- Use a load balancer/gateway between your servers and the outside world (only port 80 and 443 should be open, 80 should redirect to 443).

- Use Docker-(Compose) as runtime/orchestrator only for development. For production use hardened cloud runtimes (e.g. Google Cloud Run, k3s/k8s, ...) with zero trust as default setting. Adhere to best practices!

- Before migrating production data, make sure everything works and nothing is exposed to the webs.

Tbh. I think your blog post is ridiculous, because IMHO the whole thing is your fault.

When I first read the original post, I felt inclined to shift the "blame" on the author, much like they shifted it on Docker, though they are aware of how this footgun can also be a benefit to some:

> It would make for a much more dramatic read if I was hit through a vulnerability in Docker instead of a footgun. By having Docker silently override the firewall, Docker has made it easier for developers who want to open up ports on their containers at the expense of security.

When I want to run my own web server on an arbitrary Linux distro inside of a container, I want to bind to 0.0.0.0:80 and 0.0.0.0:443. That's it.

When I want to run my own database on an arbitrary Linux distro inside of a container, without making it accessible externally, I want to just omit exposing ports. That's it.

I don't want to think about the firewall on the system (even what kind of a firewall it is running, if any) or configure it separately, otherwise it'd be as bind mount target directories on host system not being automatically created, making me use "mkdir -p PATH", which already adds unnecessary friction (if I want the files to be available in the host OS, as opposed to just in abstracted away Docker volumes). After all, if there were more steps, then an awkward dance would inevitably start: "Hey, we launched the container on SOME_DISTRO but we can't access the UI/API/whatever." and you'd have to write platform-specific instructions, as opposed to being able to just give a Docker Compose file or an equivalent and be done with it.

Thus, my original thoughts were along the lines of: "It's not at the expense of security, as much as it is to the benefit of your convenience without any changes to security, if you are aware of what you are doing." though that isn't charitable enough. Something like this would be more helpful instead:

  # Suppose you want to let people know that your container wants to listen on the port 2000
  # You can specify it in the Dockerfile, though this will be more like documentation for the container, as opposed to publishing anything
  # Docs: https://docs.docker.com/engine/reference/builder/#expose
  # File: Dockerfile
  EXPOSE 2000/tcp
  ...
  
  # If you want your port to be available locally, then Docker gives you that ability as well
  # You can even change the port, for example, to 2005 on the host
  # Docs: https://docs.docker.com/compose/compose-file/compose-file-v3/#ports
  # File: docker-compose.yml
  ports:
    - "127.0.0.1:2005:2000/tcp"
  ...
  
  # If you want your port to be available remotely, then you can either omit the IP address or use 0.0.0.0
  # Let's take the above example and use 2005 as the exposed port again
  # File docker-compose.yml
  ports:
    - "2005:2000/tcp"
  ...
  
  # For examples of Docker CLI without Docker Compose/Swarm, have a look at the docs: https://docs.docker.com/config/containers/container-networking/
  # There's also good information there about additional networking options, such as creating custom networks for limiting what can talk to what.

Actually, here's an example that anyone can run, with Docker CLI (using the httpd web server as example, same idea applies to DBs):

  # Launch everything (detached, so we can use the same terminal session, remove containers once stopped)
  docker run -d --rm --name "net_test_not_exposed" httpd:alpine3.17
  docker run -d --rm --name "net_test_local_access" -p "127.0.0.1:2005:80/tcp" httpd:alpine3.17
  docker run -d --rm --name "net_test_public_access" -p "0.0.0.0:2006:80/tcp" httpd:alpine3.17
  docker run -d --rm --name "net_test_public_access_shorthand" -p "2007:80" httpd:alpine3.17
  
  # Check that everything is running
  docker ps
  
  # Or different formatting
  docker ps --format "{{.Names}} is {{.Status}} with ports {{.Ports}}"
  
  # Then you can use a browser to check them, or do it manually, either locally or from another node (with actual IP address)
  # 2005 will be available locally, whereas 2006 and 2007 will be available remotely
  curl http://127.0.0.1:2005
  curl http://127.0.0.1:2006
  curl http://127.0.0.1:2007
  
  # Finally, the cleanup (the --rm will get rid of the containers)
  docker kill net_test_not_exposed net_test_local_access net_test_public_access net_test_public_access_shorthand

I will say that this is a good suggestion:

> Better would be for Docker to issue a warning when it detects that the most popular firewall on Linux is active and filtering traffic to a port that Docker is about to open.

However, the thing with warnings is that they're easy to miss. Either way, I've been there - when I was learning Docker a number of years back, I left its socket open and by the morning the tiny VPS was mining crypto. Good docs help, maybe exposing things publicly by default with a port binding or even using the shorthand in the first place isn't such a good idea either.

This is why you shouldn't be using Docker in production. It's a great tool, but it's simply not designed for that kind of environment.

Edit: note I said Docker specifically, nothing about containerization.

Using "ufw" on a production server is an anti-pattern, in my opinion. You should use something like ferm instead.