Hacker News Reader

Comment by jwitthuhn | Hacker News Reader

This bit me the first time I deployed using docker.

Everything but 80/443 blocked by the firewall. "Surely it is safe to run my app server on port 8000 because no one can access that port externally."

Docker stepped in to help by making sure people could access that port.

View on HN

My favourite part in Docker is what if you have 127.0.0.1 configured as your resolver (eg you are running Unbound) then Docker helpfully configure containers to use 1.1.1.1 instead.

Thanks, my ass, and give back my time trying to diagnose [a container with no diagnostic tools, because it's a lightweight container] why it can't connect to someservice.on.my.lan.

Nobody should ever mess with a system's resolver settings.

I've recently had to debug a bug with a container (probably not Docker itself) unable to reach the internet. It turns out the machine is behind a NAT64 and has IPv6 addresses only, yet something in the container helpfully reconfigures the resolver to be 8.8.8.8 which of course is unreachable.

Ubuntu on WSL is really bad about this and you have to end up using chattr to make /etc/resolv.conf immutable to prevent it from fiddling around with it. Really makes you appreciate Slackware/FreeBSD.

Running anything on NAT64 without 464XLAT is a broken setup.

I have to admit with some embarrassment that I've never deployed an ipv6-only infrastructure, so I've never had to figure this stuff out. 25+ years doing Linux/infra and it's still arcane and avoidable. I don't even know what 464XLAT is, and have to look it up.

> 464XLAT allows private IPv4 clients to connect to an IPv4 host over an IPv6 network. It only supports client-server, not peer-to-peer communication or inbound connections.

...A brief description for anyone else who didn't know.

Wouldn't 127.0.0.1 be the loopback for the container itself? So you'd need to set the DNS to the gateway IP or dynamically by resolving `host.docker.internal` (assuming `--add-host=host.docker.internal:host-gateway`)

No, that won't work if the resolver is listening on 127.0.0.1, which will most likely be the case.

Docker also has special handling for systemd-resolved: if resolv.conf points at 127.0.0.53, it will use the managed resolv.conf located in /run.

Yes.

> So you'd need to set the DNS to the gateway IP

This is the least friction way. Run your own resolver there to resolve names for containers, forward to the host's loopback for everything else.

> or dynamically by resolving `host.docker.internal`

Resolving. Name. Of the DNS server. Right?

I mean dynamically as in a startup script that resolves that hostname, then updates the resolver to the IP it returns.

Doesn't even have to be dynamic, Docker engine knows the IP of the host, so it can just directly configure `/etc/resolv.conf` with it.

Might not bind on the external interface though.

Ah.

This would require adding a script to the image itself, for each image. No go.

This is such scary stuff. I love the ease of docker (compose) but stuff like this makes me miss the old days. No surprises. Now you can deploy a huge stack of stuff but it may work differently from expectations.

Its scary stuff because the common theme of software dev/deployment is to follow common patterns without actually understanding what goes on under the hood.

And no one follows what happens under the hood because companies don't value it at all (until something goes wrong, at which point they fire the scapegoat)

That issue bit me with a dockerized mysql db. I think because I was not familiar with `ufw` at the time that after I set it up I tested to ensure you couldn't get to the db.

Surprise enused.

I agree this is surprising behavior. Other comments have placed blame at the feet of Docker, iptables, or Ufw. However the easiest workaround is to tell docker to publish the ports ONLY on 127.0.0.1, making them inaccessible from elsewhere:

- ports: 127.0.0.1:8000:8000

I have found that another long-term solution is to switch to rootless Docker[0] or Podman for local dev. That way iptables isn't mangled with because you don't elevate to a privileged user to use it. Not being privileged has some downsides, but I think the tradeoffs are worth it. Docker Desktop and Rancher Desktop also is great on Linux for this reason, although, they both require a desktop environment.

(Not disagreeing with your point -- yours is a great/quick workaround -- it just seems that many people don't know about rootless Docker)

[0] https://docs.docker.com/engine/security/rootless/

I don‘t get this. What‘s the point of publishing them on some port if you don‘t that to be.. well, public?

Docker lets you publish services for consumption by other services in your application. You don't necessarily want all of those services to be accessible to the whole world. Database, for example.

Interesting. I guess I've been lucky as I use externally managed stateful and stateless rules (aws security groups + network acls)

Reply on news.ycombinator.com