Hacker News Reader

Comment by eganist | Hacker News Reader

It's not all that great blaming the victim. They clearly made the right moves with at least some of their configuration decisions and leaned on the underlying platform not being bonkers (but alas, it was: https://github.com/moby/moby/issues/4737 and https://github.com/moby/moby/issues/22054).

Should they have hardened in all the other ways for defense in depth, e.g requiring authentication from localhost? Sure. Should the attacker have not done the attack? Without a doubt. But should docker have not completely undone a security control?

Yes, and the fact that docker seems to have persisted with the current state is the topic of discussion.

Previously: https://news.ycombinator.com/item?id=27613217

View on HN

This same one got me after years of using Docker, I only discovered it after using the combination of Ubuntu Server (and it's ufw) on a DMZed device.

I was running what I thought was an internal FTP instance for almost a week. Luckily it was about as hardened as regular ftp can be, but I noticed the problem when my service wasn't able to log in as the (very low) connection limit was filled by someone attempting passwords.

I've been using https://github.com/shinebayar-g/ufw-docker-automated to make docker compliant with ufw, and defining firewall rules as labels for the containers. It needs some work still, namely the service should be hosted in it's own container for easier updating, but it works reasonably well.

Yep. This exact interaction between UFW and Docker had me stumped for a while too.

Totally unexpected outcome.

Luckily I discovered it in testing, so didn't make it to production. But annoying that those issues still remain for so long.

that first issue links to a really great flow-chart of how iptables works: https://cesarti.files.wordpress.com/2012/02/iptables.gif

I haven’t used docker in production. I guess it is used for k8s all around?

> It's not all that great blaming the victim

* Run a SaaS that costs money

* Do not perform the migration in prod replica setup with test data and monitor for, observe and fix any oddities

* Cause a 100% data breach by not following a sound change management procedure

Just because this individual is being blamed and not a Ltd. or Corp. doesn't mean that this is how a quality SaaS should be run. I know that more it's-just-a-side-gig SaaS wing it more than they do not but this is exactly what happens when you haven't got good ops. Rather than random error, this is a consequence from a systematic error.

> They clearly made the right moves

The machine with DB had a public interface. No matter firewalls, this is just bad. DB machine should be in private subnets, preferably with no inet access at all, even via NAT.

Proof that it had public interface. TFA: "Docker helpfully inserted an allow rule into iptables, opening up MongoDB to the world"

Even Google Cloud SQL just is a managed instance with a public IP (that you don't seem to be able to make private). Firewalls always help, and thats what Cloud SQL does: whitelist your ip when using the CLI.

What I do in AWS is having a VPC with 3 subnet types, public, private and restricted. Public ones have default route to inet, private to NAT, restricted - none. DBs go in restricted, load balancers in public and web/app servers in private subnets. Plus whitelist firewalling using Security Groups.

when i evaluated using a NAT setup using AWS NAT gateways it appeared to be quite expensive. we settled for strict VPC rules using only SecurityGroups in order to restrict communication. the instances did have public ip but nothing but intended services could connect to it anyways... i.e an instance running a HTTP API could only be connected to if initiated from the load balancer and itself could only connect to relevant services such as a RDS cluster...

Normally machines behind NAT use it only for pulling stuff on boot time. Main traffic goes via LBs. depends on your case, of course.

later on they also had the need for it to be honest, albeit only for a very specific external service that required us to have a fixed public IP address to connect from. The only solution deemed scaleable was using such an AWS provided NAT Gateway..

so yeah, it surely depends on the case given...

>The machine with DB had a public interface. No matter firewalls, this is just bad. DB machine should be in private subnets, preferably with no inet access at all, even via NAT.

This is the type of thinking that lead to the attack in the first place. No, you should not pretend that your database is secure because it is hidden deep behind dozens of layers of firewalls.

You should assume that your database is always accessible from the interne and make sure that internet access to your database is not relevant.

With mongodb you are supposed to use TLS encryption and authentication even for communication within the same host because a low privilege hack or mistake could forward a database port or make it public.

In this scenario you not only need to bypass all the network security theater, you also need the password and a valid client certificate signed by a private CA.

What he said about the DB being in a subnet wasn't incorrect, nor what you just said about setting a password and using certs. It's defense in layers and not "network security theater". You don't do just one of them.

> Published ports

> This creates a firewall rule which maps a container port to a port on the Docker host to the outside world.

Source : https://docs.docker.com/config/containers/container-networki...

Requiring authentication from localhost does not seem relevant to me, given that the creds would be stored somewhere, either in memory either in a file anyway, but exposing a port is not "binding on localhost".

However testing your firewall after publishing a docker port seems common sense.

Indeed, that's how I found out Docker was using the DOCKER-USER iptables chain that you can customize:

https://docs.docker.com/network/iptables/

And that's how I made a simple firewall that works:

https://yourlabs.io/oss/yourlabs.docker/-/blob/master/tasks/...

Another thing, instead of using exposing ports like that, the easiest is to use Docker-Compose, so that your containers of a stack have their own private shared network, then you won't have to publish ports to make your services communicate. Otherwise, just create a private network yourself and containerize your stuff in it.

So for me that's two newbie mistakes which conducted to falling for this non targeted, script kiddie attack which has been going on since 2017.

But yeah, go ahead publish your ports instead of using docker networks how you should, "believe" in your firewall while you're at it, and then blame "docker footgun".

> Another thing, instead of using exposing ports like that, the easiest is to use Docker-Compose, so that your containers of a stack have their own private shared network, then you won't have to publish ports to make your services communicate.

That only works with local communication unless you use docker swarm. So byebye high availability.

If you're going to make suggestions, at least think about them from a production point of view.

Most productions are fine with 99.9% uptime, and a single server is fine for that, so HA is not necessary for most production out there.

For example, I'm running a governmental, nationwide service handling 300k req/day, with 1k admins working daily on the site, on a single server without anyone complaining, and without less than 99.9% uptime even though I reboot to upgrade the kernel twice a month, I did have to fine tune the system but that was actually pretty easy.

That’s 10qps assuming working hours indeed you don’t need more than one server for that.

How many user facing projects actually got so far they needed more than one server? 1% of them?

I'd say even less - a single db server is almost enough for let's encrypt: https://letsencrypt.org/2021/01/21/next-gen-database-servers...

Quick calc: 99.9% of 365 means 8:45 downtime a year. That's not a lot, especially if hardware or power fails.

In my personal experience, given a vendor SLA of 4:00 hours, Dell manages to send you replacement HW on time, but HP has up to now failed to deliver in window every time. YMMV of course. RAID is the only thing keeping us online in these cases.

There is also the normal server maintenance. Rebooting a kernel twice a month, if the reboot takes 2 minutes, already swallows 48 minutes.

Now to be honest 99.9% business hours only is mostly doable if you serve only 1 continent and manage to update/reboot outside business hours. I have a non-critical debian public web server running that updates and reboots every day at 3AM. Mostly because nginx didn't consistently apply lets encrypt certs. So far no one has noticed or at least complained.

If the server burns I'll just deploy another one from scripts and restore the data, I won't need to wait 4 hours I can do it right away, I will surely be able to do that in less than 8 hours!!! However, we don't own the servers, we just rent them from OVH, so we can spawn one in less than an hour and provision it in another, so maybe the fourth time my server actually completely burns in a year I will feel some kind of fear to do 99.8 instead of 99.9 :'''( <--- these are many tears.

Funny to see my above comment get a final score of -4, people don't like us saying "well just get a cheap linux box and we'll get you 99.9% uptime" ahah and this is "hacker news"!

Not to mention the comment that was like "you can't do production without HA", I have been developing, deploying, maintaining a bunch of productions for the last 20 years and very very few of them needed any kind of HA.

Anyway, with that experience I can get you >99.9% uptime with a 3 buck/month linux box, or HA for anything above 99.9, cause that's what experience is about kids ;)

> Indeed, that's how I found out Docker was using the DOCKER-USER iptables chain that you can customize

Only on IPv4...

> Yes, and the fact that docker seems to have persisted with the current state is the topic of discussion.

It's clearly written in the docs:

    To expose a container’s internal port, an operator can start the container with the -P or -p flag. The exposed port is accessible on the host and the ports are available to any client that can reach the host.

(from https://docs.docker.com/engine/reference/run/#expose-incomin...)

It should be common knowledge by now to either create a virtual interface / network on the host to publish ports on for usage by services that aren't running in Docker or (if you are in an environment where all services are in Docker) use --link between containers.

I don't think that's clear at all. If I set "bind_ip = *" in some application then it's also "available to any client that can reach the host", but the firewall is in front of that. I certainly wouldn't expect an application to frob with my firewall.

And as I understand it, this is very much an unintentional side-effect of ufw and Docker interacting – it's not Docker's the intention at all to override any iptable rules, just an unfortunate side-effect.

"It should be common knowledge by now" is very hand-wavy. I never used Docker much, and I could have been bitten by this. Where do I get this "common knowledge" from? Not everyone is a full-time sysadmin; some people just want to run a small service, read the basic documentation on various things etc., and set things up. Not everyone is super-invested in Docker (or any other tool for that matter) to have all the "common knowledge" that all the experts have. This is why defaults matter.

Same here. It's even a little worse because of the vagaries of Docker networking. I have a server, it has a firewall that has all ports blocked. If I install Docker to run some containers and use the --network=host option, it all works fine. Containers listen on ports on the host, my existing firewall controls it, great. If I use what would seem like the more secure port-forward option, Docker adds a new rule to the top of my firewall ruleset opening the forwarded ports to all traffic, overriding my existing firewall for that port entirely.

I get that Docker is often run by devs on their laptops, but this is bad behavior. The simplest solution I've seen so-far is Docker should only automatically set these firewall rules on devices that only have RFC1918 IPs. That would still keep things simple for devs on laptops, but would prevent firewall override on servers.

You forgot IPv6, unless Docker still doesn't support that, ofc.

> "It should be common knowledge by now" is very hand-wavy.

It is written explicitly in the manual that any port you choose to publish is reachable by all machines that can reach the host.

Just how more explicit does this warning have to be?!

And anyways: a simple "netstat -lnp" would have shown the docker-proxy process and world-wide reachability.

What would you estimate as the total word count of the Docker documentation? The page you linked to alone is about 10k words, and there are a lot of pages. At a guess, we're looking at something the length of notoriously long novels like War and Peace. What percentage of the documentation had you read and internalized before you first put Docker into production?

It's very easy to come along after something blows up, find something a person could have done differently, and blame that person, treating them as stupid or negligent. It's easy, satisfying, and often status-enhancing. It's also going to make the world less safe, because it prevents us from solving the actual problems.

Those serious about reducing failures should read Dekker's "Field Guide to Understanding 'Human Error'": https://www.amazon.com/Field-Guide-Understanding-Human-Error...

It comes out of the world of airplane accident investigations. But so much of it is applicable to software. The first chapter alone, the one contrasting old and new views, can be enough for a lot of people. It's available for free via "look inside this book" or the Kindle sample.

> What percentage of the documentation had you read and internalized before you first put Docker into production?

Admittedly, not as much (and since I've got a couple of years experience, back then the docs were bananaware), which is why I ran into the same issue - but one thing I always do is set up staging environments and check if at least the basic expectations (=can't reach an internal service from the Internet, but can reach public services) are working. Aka: a portscan.

A basic check would have prevented so many carelessly-configured services. Generally, it doesn't hurt anyone to run a nmap scan as part of the regular monitoring - if the line count doesn't match the previous one, raise an alarm.

Oh, is that also in the docs?

My point is that this is the same behavior: looking for a reason why the person is wrong. That is a behavior that makes things worse because it takes the focus away from systemic improvements that make us safer. Please don't do it.

> That is a behavior that makes things worse because it takes the focus away from systemic improvements that make us safer.

Systemic improvement would be to educate people to build fail-resilient systems. There are a number of reasonable precautions that could have prevented all of these "I exposed my unsecured MongoDB docker to the Internet" instances:

1) Dedicated hardware firewalls (or in the cloud, firewall rules): If all your server is accessible on is 80/443 anyway from the Internet, it doesn't matter if you punch a hole in the server's firewall.

2) Regular outside port scanning as part of your monitoring. This helps you catch misconfigurations of your firewall.

3) When planning a new infrastructure, think about which system needs access to which other systems. Your MongoDB doesn't need to have an Internet connection

4) Have someone experienced look over your staging system, do a full scale security audit or attempt basic measures of pentesting on your own.

5) Don't copy shit from stackoverflow or years old random blog post without understanding every single thing it does.

6) Don't use MongoDB which has been so fucking often in the news for being an insecure pile of dog poo that it physically hurts to read that name!

Organizations that fail to do at least the first four points as part of standard operating procedures deserve to be hacked, period. There is no excuse for slacking off on either of these in 2021.

While common knowledge, the resolution is not always clear. Is the current expectation that everyone using docker is schooled in network engineering?

> It's not all that great blaming the victim.

Everyone wants to be treated as a (software) engineer here, but the engineer's perspective in this situation would be the opposite: The victims are the customers, and the perpetrator, acting in negligence, was Newsblur. Risk management is a core part of the engineer's job. https://www.sebokwiki.org/wiki/Risk_Management

> Everyone wants to be treated as a (software) engineer here, but the engineer's perspective in this situation would be the opposite: The victims are the customers, and the perpetrator, acting in negligence, was Newsblur.

There can be multiple victims, multiple causes/threat actors, and overlap between two categories.

But what's the end user a victim of in this case? Other than a brief lack of availability?

Is there any other profession were the professional, doing something wrong, would be called the victim? Would an engineer, having made an error when doing structural engineering calculations, be called a victim if the software used was too complex? Or a physician who overlooked adverse affects written in fine print? Or the electrician making errors when fixing things in a house with old wiring?

Of course the behavior of Docker is really problematic in this case, and ditto for MongoDB not having auth active per default. But both applications were consciously chosen by Newsblur.

> Is there any other profession were the professional, doing something wrong, would be called the victim? Would an engineer, having made an error when doing structural engineering calculations, be called a victim if the software used was too complex? Or a physician who overlooked adverse affects written in fine print? Or the electrician making errors when fixing things in a house with old wiring?

I don't understand the relevance of this to my original question.

> Of course the behavior of Docker is really problematic in this case, and ditto for MongoDB not having auth active per default. But both applications were consciously chosen by Newsblur.

Or this, for that matter

---

So that you have it, my original ask was: But what's the end user a victim of in this case? Other than a brief lack of availability?

Your actual original point was:

> It's not all that great blaming the victim.

Which is what I was replying to.

Of course this incident was inconsequential for Newsblur's customers. That in this case the impact for the customers was approx. zero doesn't change the relationship of Mongo+Docker <> Newsblur <> customers (i.e. who has responsibility and who feels the consequences, which defines who can be a victim), just the outcome for the customers.

> Of course this incident was inconsequential for Newsblur's customers. That in this case the impact for the customers was approx. zero doesn't change the relationship of Mongo+Docker <> Newsblur <> customers (i.e. who has responsibility and who feels the consequences, which defines who can be a victim), just the outcome for the customers.

See my previous note:

> There can be multiple victims, multiple causes/threat actors, and overlap between two categories.

I think we've both concluded now that newsblur was largely a victim as a consumer of docker.

Cheers, glad this helped.

If there is a third-party exploiting that error to cause damage, sure (not the case in your examples, which is why they miss the point). If you don't get security for your business right and it gets burned down by an arsonist, nobody is going to not consider the business owner a victim too.

Software culture is IMHO problematic there in multiple dimensions.

This is completely wrong. By this logic me killing a parent turns the parent into a perpetrator because the parent didn't learn proper self defense.

No, the parent is a victim and the children are dependents of the victim. Thus hurting the parent also hurts the children. It's the same thing with Newsblur. Hurting Newsblur hurts their customers.

Reply on news.ycombinator.com