Show HN: Bunkerized-Nginx – Nginx Docker image secure by default | Hacker News Reader

Show HN: Bunkerized-Nginx – Nginx Docker image secure by default

github.com

221 pointsbunkerity88 comments

Archive.org's archiver is a "bad user agent" according to the list it uses, apparently: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultim...

It can be disabled (BLOCK_USER_AGENT=no), but it is on by default.

Why would someone like to block crawlers and also TOR-clients? The malicious ones will be anyway disguised as normal browsers.

Considering they quite explicitly ignore the nice way to say "don't archive me" (robots), yes, it is a bad user agent.

considering how many sites say "dont archive me" while residing as a public resource on the internet at the same time, i support this practical ingnorance.

I always think it's funny that a community that hates the AGPL for being too restrictive to corporations cause they definitely can't be expected to share their code in exchange for whatever they're taking also wants all takes, good or bad, that anyone has ever puts on the internet to be preserved.

For folks interested in further reading, I like this guide for hardening nginx: https://github.com/trimstray/nginx-admins-handbook

Seems some very critical security mechanisms (such as CSP) still require additional/careful user config, and don't get highlighted by the guides. Might help to add a walk-through or, e.g. link to a good CSP generator tool.

Also I wonder if compiling from source then using `FROM scratch` would improve security even further, similar to this nginx container image: https://github.com/ricardbejarano/nginx

I've used browser extension last time:

https://chrome.google.com/webstore/search/csp?hl=en-US

Helpful to gather all the assets being loaded and generating the rule sets, it's probably better to work on dynamic websites which may load additional data live.

I had a lot of Google Tag Manager and 3rd party libraries.

From what I understand, when a user completes a captcha, you set a session cookie, using resty.session, which I assume is this library: https://github.com/bungle/lua-resty-session

According to its readme, it does not check IP addresses, which means an attacker can just complete the CAPTCHA once, and copy their session to as many computers as they like. So you probably want to set session_check_addr.

But I've never used resty or lua, so I may be misreading how your code works.

Hundreds of configurable parameters, thousands of lines of code ...

Are you sure that this is actually more secure?

Hmm, looks like it would serve a very similar role as Caddy does for me right now (though my Caddy SSO w/ JWT is something I'd have to recreate). I wish this had existed a few years ago and I'll absolutely be bookmarking this for future projects. Very cool!

Also using Caddy as a reverse proxy for a number of containers. This is definitely an attractive alternative.

Really isn’t of concern at my current scale, but Nginx offers some margin of speed over Caddy.

Agreed, I’ll probably keep using Caddy for my self-hosted apps but this looks like a nice alternative for more production-facing things.

Can you tell us more about your Caddy+SSO setup?

Sure! I have all my self-hosted apps running on subdomains under 1 domain. So let's say it's "mydomain.com" and then I have subdomains like "gitlab.mydomain.com". So for this setup I add a block at the top of my Caddyfile:

    (secure) {
      jwt {
        path /
        redirect https://auth.mydomain.com/login?backTo=https://{host}{uri}
        except /api
        except /rest
        except /zm/cgi-bin/nph-zms
        except /zm/api
      }
    }

The "except"'s are for some services I use that have some kind of api-based auth for reaching certain endpoints that need to be whitelisted. This approach (of importing this "secure" block in each caddy block, that's just what I called it, "secure" isn't a special word) has the downside of requiring a global list but makes it easier as you will see in just a minute. Also there wasn't any overlap of "services that have api endpoints using api keys" and "services that have the same endpoint but it needs to be under SSO auth". Moving on, after that block I have my "auth" url, again you can name this whatever:

    auth.mydomain.com {
      login {
          simple username=password
          jwt_expiry 24h
          redirect_check_referer false
          redirect_host_file /root/.caddy/hosts
          cookie_expiry 2400h
          cookie_domain mydomain.com
        }
    }

The "simple username=password" is the line doing a lot of the heavy lifting but you can replace that with something that uses a different SSO provider (like Google, LDAP, etc). I have a random U/P that I store in 1Password so "simple" is fine for me but I've wanted to setup Google's OAUTH to at least test it out. The other big thing here is "/root/.caddy/hosts", this is a list of all the domains/hosts (one per line) that you want to be able to redirect to. This is so that if you go to "gitlab.mydomain.com" and you aren't logged in it will bounce to auth.mydomain.com and then, because you put "gitlab.mydomain.com" in that "/root/.caddy/hosts" file, it will redirect back to "gitlab.mydomain.com" once you login. Gitlab may be a bad example since you will absolutely be using GL's auth as well but substitute Gitlab with other services that either don't provide auth or provide some basic auth you can turn on/off, that's where this really shines. Once you are logged into 1 of them you are logged into all of them.

Lastly we have our actual, regular, caddy entires: (and looking at mine I see I should have been using Syncthing as the example all along haha)

    syncthing.mydomain.com {
      import secure
      proxy / 10.0.1.123:8384 {
        transparent
        websocket
      }
      gzip
      tls myname@mydomain.com
    }

I will note that you might not need the "websocket" line anymore but it works and I'm not touching it. That "import secure" line is what "protects" the Syncthing service.

It has been a breeze to add/remove services that I want to stick behind auth and I'm very happy with it. A couple caveats: I still need to update to Caddy 2, it came out shortly after I did all this work I decided to sit back and wait for the dust to settle. Also I wasn't able to use the default caddy docker image, it needs some extra plugins. This may no longer be the case for Caddy 2, I don't know. I ended up just making my own image [0] (/Do not use this/, really, don't. I'm not going to keep it updated, I make no promises, and it's a huge security risk IMHO) by forking the repo and adding the extra plugins I needed [1]. You can do the same and build locally or maybe you don't even use docker and so this is a non-issue. The two plugins I needed were "jwt" and "login".

I hope this helps and I can answer any other questions you might have about it.

[0] https://hub.docker.com/r/joshstrange/caddy

[1] https://github.com/joshstrange/caddy-docker/blob/master/Dock...

that's just what I called it, "secure" isn't a special word

On a slight tangent, this is a good thing to look out for when writing docs or other explanatory notes that include code/config samples - if your sample includes names that could be confused for some semantically meaningful thing, change the name to disambiguate or mention it in the sample explanation, like above.

This is very cool, thanks for the explanation.

Have you looked into how to migrate this to caddy v2?

Not him, but this should be quite easy to port, the only difference being that the JWT stuff would need to be served by this newer plugin instead https://github.com/greenpau/caddy-auth-jwt because v1 and v2 plugins are not compatible.

  I wrote a whole blog post [0] about this that just completely slipped my mind. I think my comment does a good job of getting the same points across but I can’t believe I spent 20-some minutes regurgitating this information haha.

[0] https://joshstrange.com/securing-your-self-hosted-apps-with-...

> Block TOR, proxies, bad user-agents, countries

Does this really help? Especially blocking TOR and whole countries seems bad. As for blocking user-agents: why? Sorta seems like when cisco "fixed" a vuln by blocking the curl user agent.

As with most things, I would say that it depends on the use-case.

For example, this image might be good for someone who wants to host an internet facing service but only for a few friends, for example a some sort of git forge.

Why wouldn't you use normal auth for that? or client certificates if you don't want to handle passwords/SSO?

How does blocking user-agents, countries, TOR or proxies help with hosting a git forge for a few friends?

I have a Digital Ocean droplet that I use to host personal sites and projects. I've had it for 3-4 years now.

Every morning I get an email from logwatch showing me activity on the server, particularly failed http requests and "attempts to use known hacks" (which are just pattern matches on URL requests).

I used to get hundreds of lines of logs PER DAY of this stuff..

Then about a year ago I found a list of IP subnets for China and Russia, and blocked all of it at the OS level with the firewall, and now I get almost nothing in my logs anymore (every so often I add a new subnet to the blocklist, usually from eastern Europe).

I can't see any reason why my server that hosts personal stuff would need to see traffic from those countries (and I'm fine with blocking false positives - YMMV).

I know it's not a foolproof security measure and if anyone was specifically targeting me this would not stop them, but it helps with all the drive-by scanning.

There is a TON of automated scan traffic coming out of China and Russia specifically that is constantly scanning IP blocks known to belong to hosting providers.

But why do you care about failed login attempts? If your passwords/keys are good enough and you keep your software updated why care about failed logins?

You misunderstand, these are not failed logins.

These are web scanners trying to find common vulnerabilities in typical web software (phpMyAdmin, Wordpress, Rails, etc). But also potentially exploits in nginx or other parts of the stack for all I know.

And they send requests by the hundreds, which pollutes my logs and my logwatch emails.

I have automatic security updates turned on for my server, but it's not my full-time job to manage it, so I want to maximize the return on the time I do spend managing it.

And since there is zero value/reason in allowing access to my server by anyone in those countries, why not just block it all and have one less thing to worry about, since 99% of that automated traffic comes from there?

Prevent 99,9+% of bots attacking with random vulnerabilities or brute forcing your private web software? I see it as a defense-in-depth mechanism.

If your software can be successfully hacked by "random vulnerabilities" or your passwords can be that easily brute-forced then it's not going to help anyway, and if they can't then I don't think you should care that someone failed a login.

or if they pay the network usage.

What other ways of blocking users do you know? Imagine you have a website with your local town adverts and spammers manually keep registering to your side, confirm email and then keep account appearing to be used by a true user and suddenly start spamming. You block one account, find a couple similar then block them too. Then you get a complain from a user claiming to own one of those accounts. Let's say you ignore that. Few days later a new spam account "wakes up". You block couple similar accounts. Tell spam detector to block certain sentences and domains. Another complaint. Then another account "wake up", this time spam filter catches it, then spammer tries manually changing their spam until it goes through. You block those too and delete accounts. More complaints. Now other legitimate user complain that they cannot share a link or write certain sentences. Turns out spam filter is too trigger happy. You spend time to tune it. More accounts "wake up". This goes on and on. You find yourself no longer working on the site, but firefighting spam. You report the IP addresses used for spam. Nothing changes. Then you find out the IPs are from a couple of countries, that are not your town. You block the IPs. You get complain legitimate user on holiday can't access it. More accounts "wake up" - from new IP addresses. It is TOR. You block TOR. More complaints that some users cannot access, but other than that no more complaints about spam. Finally you can go back to developing your product. It is simply not worth. I advice to block everything from various lists as the minimum. Countries don't allow people from dodgy places, websites should be able to do too. People should be going to jail for spam. They damage economy. Big companies have resources to deal with it so they don't lobby to make law changes as it takes potential competition at bay.

> You block TOR. More complaints that some users cannot access, but other than that no more complaints about spam.

Sure, if this worked. But it's not 2010 any more. Simple stuff like this only stops the most technologically illiterate attacks.

If your site is being attacked by more than bubba the local "computer guy" - they will be using proxies indistinguishable from your normal traffic, assuming you do any volume at all.

It's just a silly exercise in futility. I put it up there with moving sshd from port 22 to a random port. Yeah, you get less spam I suppose, but it sure as hell didn't make you any more secure. Probably the opposite.

Outright blocks of "bad IPs" are not that useful in my opinion. However, developing reputation systems around IPs and using "is tor" as one metric certainly has shown it's uses in my field of work.

> Simple stuff like this only stops the most technologically illiterate attacks.

If 90% of spam that requires human moderation comes from those, this seems like an excellent starting point though, as indeed is moving your ssh port. Security is in layers.

Most attacks in my experience over the last 15 years are exactly that, though: simple uncomplicated ones

Ok so you bashed the method I used and you have not provided any examples how this could be done better. That is unhelpful.

Expect blocking parts of the internet does nothing to help with this. You deal with automated spam on a website by restricting who can post or by doing automated human verification like CAPTCHA and having spam filtering.

Just blocking parts of the internet does nothing to help the problem and simply introduces other problems.

These bots were operated by humans. I tried many different captchas and that didn't stop attacks. In some parts of the world it is cheaper to get a human than the IP address. If you block addresses they have access to, it suddenly becomes not profitable to spam, because the IPs you have to get are too expensive.

Blocking parts of the Internet absolutely helps the problem. It doesn't _solve_ the problem, but it can reduce the amount of effort you spend dealing with it.

Time is money, and for most businesses or people, this is the most important metric. If I can spend 5 minutes on an 80% solution, or 5 days on a 90% solution, I will absolutely start with the 80% solution and see how things develop.

A more secure (and much simpler to maintain) version would be just running nginx from your repos without all these extra things. nginx is secure by default. The insecurity comes when you pile on all the dynamic and container stuff.

I'd prefer if this were not offered as a docker image, but rather as a configuration file or config file generator, akin to what mozilla does for recommended SSL settings [0].

A config generator seems obviously better since anyone who knows what they're doing can use it with either nginx installed from their distro's repos, or with an nginx docker container.

[0]: https://ssl-config.mozilla.org/#server=nginx&version=1.17.7

While it's true that nginx is secure by default, this image adds extra best-practice security addons on top of the out-of-the-box security. Rate limiting, fail2ban, DNSBL is all stuff that you'd otherwise have to configure manually but this sets it up for you.

Wouldn't you generally want all that stuff on the Docker host though?

EDIT: or running as separate containers on the host - the point is that these are security provisions you want to apply across all containers, not just for Nginx.

OK. Just as I thought I understood the Docker world view...

Surely all the complexity should be inside the container and the surrounding host should be as vanilla as possible.

Isn't the idea to push the complexity into the layer where builds are reproducible and state is controllable?

I am probably misunderstanding something here. I'm fairly new to this.

It violates a lot of principles of using containers. There's no separation of concerns here, builds aren't reproducible, it hijacks logging for its own setup, appears to allow RCE of php files on another host in the event it is compromised (does it make sense to store/bundle php files in this container if it's not meant to handle php?)... Etc

Sorry - I'm not clear what the referent of the word "it" is here? You mean Bunkerized is violating these principles? Or the approach the gp suggests?

The linked repo. I mostly agree with you that most of this should be in containers; I should have been more clear to what part of your comment I was addressing.

In regards to docker worldview, this project currently doesn't follow best practices.

And while I agree mostly with this statement:

> Surely all the complexity should be inside the container

The caveat being that complexity should be split up into separate concerns. Otherwise there's little difference between the host and container aside from an extra layer of abstraction.

For example, this repo should probably be split into several containers: cert management should probably be its own container, which a shared volume for certs); php should be rolled into its own container, and php files should reside there; logging shouldn't be handled at the container level; firewall concerns (namely fail2ban) probably should be handled at by the host, or in a container with appropriate permissions; etc

You're correct and absolutely want it all wrapped up in a container (or multiple containers, depending on preferences).

And yes, you want the host as vanilla as possible. Generally just running the orchestration layer (Kubernetes, Swarm, Nomad) and maybe some logging / metrics stuff (and, depending on who you ask, sshd).

You’re correct. All the traffic should be routed to the nginx endpoint and the host should be vanilla.

I don't agree (based on the fact that the host can run multiple images/VMs). In my opinion first of all the host should be secured (Firewall & Fail2ban etc...).

To distribute security to the single images/VMs increases complexity and the likeliness that some image/VM will miss some security filter, and leaves the host itself unprotected (e.g. network time sync & ssh & other stuff will probably be running, any update to the host's SW might result in unexpected services running, etc...).

An additional (dedicated) layer of security in the images/VMs would of course still be ok.

But you can also use a container as first contact and redirect to other containers. You can bind a network device to a container.

For example a reverse proxy container which redirects to a gitea container or a wordpress container depending on the request. The reverse proxy container can also centralize the security with certificate handling or fail2ban.

You still need access to the host, via ssh for example, to start the containers and do some basic maintenance. Won't you have fail2ban installed on the host since your ssh port would be open?

If you need direct access to the host, it’s probably a non production environment or you’re doing containers wrong. Kubernetes clusters provisioned with Terraform, for example, should almost never require ssh access to workers nodes.

This is overkill for 90% of projects out there.

I believe the conventional wisdom is for one process per container.

That is pretty much what Docker enforces,too (so it's baked in, not just happens to be best practice). You can run multiple processes in a container, e.g. using `&`, but that gets ugly so fast you'll notice you're trying to square the circle.

No, it is not baked in.

Did you know the linux kernel requires you to set an init? If you check what you were booted with (/proc/cmdline), it probably includes something like `init=/usr/lib/systemd/systemd`.

Docker requires you to set a single entrypoint not because it's "baked in", but because that's the way linux works: when you create a new namespace, you run a single program as its pid1, just like when you boot linux, a single program is pid1.

It's very easy to set pid1 of a docker container to runit or various other process runners to run multiple processes.

Arguing otherwise is no different than arguing that linux only supports running one process because if you set "init=/bin/bash" at the cmdline, it gets ugly real fast to run a multi-process system off it.

Bash is a user interface, not a service runner. If tini could run several processes, what can possibly go wrong?

I think the assumption is that nginx is your frontend server / load balancer, so all external requests go through it.

I'd want either in docker for host, or network/cdn/elb/etc if not on host

Rate limiting what? Banning what login process to nginx with fail2ban? And what's "weird behavior"? Blocking what groups from access? Users are uploading files and you need to virus scan them?

These are problems that don't actually exist by default for most situations. They imply a need for, well, something like Bunkerized-Nginx. But the need necessarily implies something less secure than a simple repo nginx install. In most cases this is the tail wagging the dog.

From the list of features:

> Block TOR

Why the hell should I block TOR? I better block users from mass testing passwords on my login masks, but do not distinguish them of their origin.

I feel that this is something that Cloudflare has had a major hand in.

Back in the day, they claimed (stupidly) that tor users where rife with DDoS attacks. This, of course, was a naked money grab by CF to scare non-technical site owners into use CF to block nasty tor users.

Also, during the (still ongoing) data collection hype train, site-owners like the idea of being able to individually identify every user, and fuck everyone else ... as is their right by Tech Decree.

So now we are at a place where just caring about your privacy, despite all of the overwhelming evidence that this is exactly what we should be doing, is prima facie proof that you are a HaX0r or, even worse, you are not exploitable for databux!

Shouldn't the Dockerfile have a USER directive, so it doesn't use root by default?

Exactly. Just like the nginx-unprivileged https://github.com/nginxinc/docker-nginx-unprivileged

I've opened them an issue just in case https://github.com/bunkerity/bunkerized-nginx/issues/11

AFAIK, it works like this:

A web server need root privileges in order to bind ports 80/443.

But in this case it’s only the primary Nginx process that will run as root. The subprocesses will run as a non-privileged user, as specified with the “user nginx” directive.

https://github.com/bunkerity/bunkerized-nginx/blob/master/co...

https://unix.stackexchange.com/questions/134301/why-does-ngi...

It's better to run as a non-root user, and use the CAP_NET_BIND_SERVICE capability (see capabilities(7)) to allow only the minimal privileges necessary to bind to a low-numbered port. A lot of Linux admins are unaware that this is the modern best practice for running applications securely.

Starting as root for the sole purpose of binding to a low-numbered port and then dropping privileges is an outdated practice that is both difficult to program correctly and arguably unnecessary today.

Thanks for pointing this out. I usually rely on the default settings in Arch Linux to handle stuff like this in the default package configuration, but it seems that the nginx package’s [1] systemd service [2] is not set up that way. Maybe it should be updated with a non-root User= and Capability=CAP_NET_BIND_SERVICE directive?

[1] https://www.archlinux.org/packages/extra/x86_64/nginx/

[2] https://github.com/archlinux/svntogit-packages/blob/packages...

This is why i love HN. Had no idea about that capability. Thanks for enlightening us!

A quick googling tells me that FreeBSD has something similar: https://gist.github.com/TomHetmer/b0a048d688af78e78f45609880...

PS. If I would have bothered to read the whole Stack Exchange article I linked to, that capability is even mentioned there (^_^) https://unix.stackexchange.com/a/134324/68449

Nice to know. Also, using a container, the port number generally doesn't matter so much.

Totally, just run things on 8080/8443 in the container and then use your privileges outside the container to map 80/443 on the host. Don’t use low-numbered ports until you truly have to, IMO.

Why not use gosu or su-exec to run the entrypoint command? That's straightforward.

Better to never have root privileges anywhere in the container. Entry points can be overridden and never running as root at all greatly reduces the attack surface.

> Block TOR

sadface

At least have a good reason for doing this instead of outright doing it for 'Security'.

As far as I'm concerned you've made the site less privacy friendly. At least redirect them to an onion of indded are shaving serious problem at some point in the Future.

Identity is the bread and butter or mass Surveillance. The two can exist with spam protection.

This problem might need a good solution to be thought of that aren't bans and extemely hard capthaa in every single page (cloudfares half baked 'solution')

That's basically what most of these "security" solutions are really all about -- about disrespecting your privacy, having no regard for inclusivity, CPU time and user time.

It's kind of sad that they've taken over the "security" moniker, and the general compute industry is totally fine with them doing so.

How do you detect anonymous spammers? Even with AI it is very difficult and some spam goes through because model didn't learn yet.

If you run software directly from git clones (without any verification steps), calling yourself secure is just a joke.

...not to mention running docker...

Keeping systems lean, with little possibilities for configuration errors, can help keep them secure. This Bloatware certainly not.

If you only read the HN comments, you will probably not check this repo out. Well you should:

- everything is easily configurable by env vars so the features that some are hating are on demand

- it is open source, so you can fork it and make your on version. You can even contribute back to improve it instead of posting hate comments on HN

Why use Alpine? A better base image would be Red Hat UBI

Why? Alpine is the de-facto standard for minimal containers. I had never heard of Red Hat UBI before, what does it bring to the table?

Alpine uses musl, which has on a few occasions led to weird and hard to reproduce bugs. Even though using musl saves a few megabytes of bloat, IMO it's not worth the headaches. Especially since some of the "bloat" in glibc is highly performant inline assembly. Plus if you're consistent about using an alternative base imagine like the slim debian/ubuntu ones, the extra storage should be more or less a constant

Alpine's DNS resolver behaves differently than glibc (in particular it ignores the `ndots` option) which can lead to DNS query amplification. In Kubernetes clusters this can be taxing on the kube-dns service and lead to cluster reliability issues around DNS lookups.

Correcting myself above: This is not the problem; the problem is that musl-libc (1) does parallel queries which can be problematic under certain circumstances; and (2) handles NXDOMAIN/NODATA responses differently than glibc does which can lead to DNS resolution failures that wouldn't happen otherwise. There's a workaround to set `ndots:1` but that causes other problems with musl since it doesn't append search domains on retries. It's just a big mess.

There's big discussion of it here: https://github.com/kubernetes/kubernetes/issues/33554

That sounds like a bug that should be reported to the alpine docker image maintainers. If not already that is...

Alpine produces minimal images, that is correct. But RedHat is a lot faster on releasing security fixes.

Because Red Hat is way faster and fixing security issues as it's based off RHEL

Docker in production nullify all the benefits of nginx (optimisations).

How so? Can you elaborate?

Knowing right principles spares one from memorising all the details.

Nginx is highly optimized zero-copy sockets-reuse code, while docker's network code is crappy, does NAT and other stuff.

I would classify it as unnecessary, redundant set of abstractions. Definitely for production.

Reply on news.ycombinator.com