Ask HN: How do you handle security for your website(s)? | Hacker News Reader

Ask HN: How do you handle security for your website(s)?

46 pointsWhiteSource130 comments

View on HN

pov of sysadmin:

- reverse proxy to backend webserver

- run checks on each request on the reverse proxy

- close all ports except what needs to be open

- only allow access to what needs to be seen

- perform frequent security updates (script it, automate it, run it hourly)

- never install more software then strictly required

- never allow SSH directly to your public address

- for ssh, use custom ports, cert (4096b) with passwords

- install logwatch or something else that will inform you about logs (elk stack)

- automate checks in logs for tell-tale signs of scriptkiddies (attempted access to /wp-admin or phpmyadmin for example) => automate adding source IP's to firewalls

- frequently check for malicious code (clamav, rkhunter, ...)

- only use stable and tested software

- only use stable and tested OS (Centos/redhat. DO NOT run production on anything else, especially ubuntu.)

- do not use ubuntu for production. it is not suited.

- install failtoban

from dev pov:

- implement measures to prevent sql injection

- don't require production to install dev libraries

- don't require production to install packages only available on ubuntu

- log strange behaviour to a file, make sure actions can be automated (iptable drop)

I'm no dev, but those are things I'd like dev's to do with their software.

> - reverse proxy to backend webserver

Can you give more clarification? Do you mean LB -> Firewall -> Web Server?

> - run checks on each request on the reverse proxy

Can you give more information on how to do this please.

Thanks

> - reverse proxy to backend webserver

Using a reverse proxy allows you to cache static content, decreasing load. I'm not talking about load balancers. These days you need them when your website pulls _a lot_ of requests (1000/s) Your reverse proxy will also stop unwanted requests and indeed will have a firewall incorperated.

> - run checks on each request on the reverse proxy

post => clamav

post => sqlinjection

get => blacklisted address?

any => iptables (let any connection go through your firewall. It's a check)

don't terminate SSL on reverse proxy though. It's a bad idea, unless you are a bad person and like to read your visitors data they think is secure.

There are plenty of other ways of using security on a reverse proxy. Have a look around on google.

No backups? Security involves the ability to recover from a breach too! :)

if you wanna go there. I use a custom script to make backups, encrypt them using gpg and store them on public ftp boxes. Automated retrieval of said backup in case of problems and automated server recovery in case of disaster.

Curious if you're not a dev how do you know this (no offense meant).

Surprised about Ubuntu, what about Debian? Share same... Hmm...

A lot of these I have to tick off.

As a sysadmin, you'd have to have first-hand experience on what can go wrong on the dev side.

Also curious as to why Ubuntu isn't suitable for production.

Maybe I have been lucky I am using nginx now on top of centos for one site but the reverse proxy for example not using that.

I am on an Amazon EC2 in this particular case, also curious that Apache was also on there but only nginx is running. Had a long TTFB was trying to figure out what was wrong. Upon restart (new instance) I guess I got lucky, now back to regular site speed around 2 seconds load (full page).

Sorry rambling, cool stuff. Been on VPS, I guess EC2 is still the same thing but next would be serverless I guess.

I am a sysadmin with approx 15 years of experience running linux systems and open source software. When I set things up, I tend to form things my way. I usually have little resources for monitoring or staff, so automation is the best to do. A server serves, it may produce errors and you can act upon that (error 404 for not finding phpmyadmin (which should _never_ be public => add ip to drop table). There are scripts that can do this for you.

However, the application which is being served, may produce different errors (failed login attempt, attempt for sql injection, cross site attack, ...). Some things are better logged by the application, Then the application needs to produce these logs. An awful lot of applications do nothing when presented with any of these type of attacks, making them extremely vulnerable. It's the dev's job to provide logging.

Debian: fine if you like pain :) it's stable.

How do you make PHPMyAdmin not public aside from say using one of those directory passwords.

Also it is somewhat of a similar question as to how Git is public but private.(Not GitHub your own setup)

There are lots of possibilities. For example: Only allow web connections through localhost and start an x session through ssh. This would require you to install a webbrowser though, so that's another security risk. Perhaps you could set up a VM with a dedi IP and only allow connections from there. (secure it well)

Another possibility would be to first pass a reverse proxy that will request a client-side certificate. That'd make it pretty secure.

A password protected directory is just asking for a brute force through a botnet. Your failtoban won't even block anything because the attacker would only let 1 host attempt a password once per 30 min or per hour. But if he has thousands of bots to his disposal, he can still bruteforce a couple times a second.

That's OK you say. I have a strong password. Don't forget: Each connection and failed login attempts is logged. Your logfile will quickly grow. If you did not set a seperate VAR partition on your disk, your root disk will quickly get full and server will crash. Additionally, it will fill up your bandwith and be as effective as a ddos attack.

Don't use passwords. Use certificates. easier for you, impossible to hack.

There are other methods of course, but these would be my preferred methods. Setting phpmyadmin publically is asking to get your data stolen. Do not assume there are not exploits in the latest phpmyadmin. But don't forget to update it within the first 10 minutes when a security patch comes along, otherwise your data may well already be compromised.

Oh man I feel so dumb, you know despite what I can do. I guess that's the problem of trying to do everything you can only get so deep.

Use certificates? What is that not keypairs?

I logged into this Digital Ocean droplet and was surprised to see 15,000+ failed login attempts to SSH. I hadn't seen that before.

I don't know if using PHPMyAdmin is noobish. I still primarily use MySQL/Maria (only).

Thanks for the tips. I learn a lot though HN.

using certificates is getting advanced. Here you'll allow the server to request a client certificate from the visitor. The visitor will have to supply the certificate (which is 4096b if you take security serious).

Easy (read easier) to do with nginx: https://www.google.de/search?q=nginx+client+side+certificate...

It requires quite a bit of steps, but is as secure as it gets what web access is concerned.

You should not be surprised to see 15k+ failed login attempts on ssh with popular ISP's. As I said, a system is the most vulnerable when it has just been installed and a/the default root password has not yet been changed. Simply disallow password login on ssh, change the port and only allow non-root users you need to allow. I have on my systems only 1 user allowed to login, authenticated by a 4096b key. There is no way an attacker can use ssh if not using an ssh exploit. The system is updated automatically every hour. This way known exploits are very quickly taken care of.

For me, server security has been a practice over many years and it takes many years to perfect your 'secure server setup'. It's depressing how many companies do not adhere good security practices and just leave their systems unprotected. Especially mail servers.

Why is Ubuntu not suited to production?

I keep hearing it but it's starting to sound like Red Hat marketing like Macs don't get viruses.

I think it can be used in production but it's more work. Red hat and CentOS can be hardened following STIG security profile on first boot from fresh install which includes SElinux checks, firewall and service configurations, etc. I'm sure there's more to it, but reading documentation on these alone is enough to overwhelm me.

An LTS Ubuntu release should be fine, though I'd hold off on using it in production until a year or so after release (to give Canonical time to shake out bugs).

A non-LTS Ubuntu release should be avoided, since they tend to be less stable.

If you're using something in the Debian family, Debian itself is probably the much better choice unless you're looking to use Canonical's enterprise products.

In my experience:

- Ubuntu is not stable

- it is not hardened out of the box

- it can contain dangerous configuration out of the box

- it calls home

- it will tempt you to install dev packages or other packages unfit for production (because devs dev on ubuntu and they require the dev packages)

But please, don't mind me. Other people are better at explaining this. I'm biassed.

For my own sites, I do all of the things jonssons said, but I also use haproxy stick table to limit the number of errors a user may generate. I also route certain URI patterns to a tarpit.

At work, I show developers, ops and infrastructure folks these things and let them decide if they want to test and implement any of it.

If you don't use FIM there is no way in hell the above list will protect you from targeted attacks.

rkhunter does fim in a limited way. I hope OP is not talking about sites with sensitive data. ossec is usually great for storing more sensitive data.

The above list will help you from most scriptkiddy attacks, which are 99% of the attacks I have experienced. It's a good idea not to piss off anybody on the net or do it with a sufficiently obscure account.

Agreed. I would add "auditd" and "acct on" to your list.

> - only use stable and tested OS (Centos/redhat. DO NOT run production on anything else, especially ubuntu.)

I'd tend to agree about Ubuntu because... why aren't you just using Debian? I would generally opt for CentOS for a prod server, but I am now starting to think about using OpenBSD in production environments.

Ultimately it comes down to the way you configure the server, not the distro you use. The only benefit of Enterprise Linux (CentOS, Red Hat, Fedora, Oracle) over Debian out-of-the-box (that I am aware of) is the presence of SELinux, which you can still set up on Debian. All have insecure configs by default.

I will say that the live kernel patching in Ubuntu Server now is cool. I used KSplice on CentOS before it was discontinued (only on Oracle Linux now).

I don't use debian because I don't like apt. The most vulnerable moment of my systems is when the ISP sends the password in plain text over e-mail. Until the password has been changed and the system has been sanitized(), I consider it compromised.

() updated, rkhunter, clamav, ...

Not all WWW sites are corporate, of course. My personal WWW site ...

... is not an API. There's no authentication, cookies, transactions, client-side scripts, or PHP. No dynamic content is generated in response to HTTP requests. It is rather generated in response to back-end events, such as pushing a new version of a package into the package repository, which cause reconstruction of the content with redo, a simple rebuild tool. So the server software itself is a simple static HTTP server. It's a slightly improved version of Daniel J. Bernstein's publicfile, in fact.

... does not put server logs, server scripts, or anything else like those in the content tree. The only things in the content tree are content files and directories. Indeed, the content server daemon runs as one user and its logger daemon runs as another, which owns the log files with only access to their owner.

... has strictly size-capped, auto-rotated, logs with the log writer doing the rotation. There's no logrotate window of opportunity for exceeding the log size. (http://jdebp.eu./FGA/do-not-use-logrotate.html) The logs are also completely independent from other non-server logs.

... runs on a dedicated machine that does not have other, non-public, services. There's no unnecessary MySQL server running on it, for example. Nor a mail system.

... is also mirrored over FTP and GOPHER. The FTP server is again an improved version of Daniel J. Bernstein's publicfile, and the GOPHER server is my addition to the same. They don't implement non-anonymous FTP, FTP uploads or filesystem modifications of any sort, or GOPHER searches, and work from (solely) the same content tree as the HTTP server. The GOPHER menus are built by the content rebuild subsystem, not by the GOPHER server.

Simple. Static HTML.

This is not what you are looking for, but this is a great checklist to make sure your site is secured. https://github.com/FallibleInc/security-guide-for-developers...

- Transparency is a dependency of trust; if it's not transparent (e.g. open source), then it's not trustworthy, full stop

- Read the latest NIST guidelines on software security; whether or not you trust the U.S. government (you shouldn't), NIST still has very solid security advice, and a lot of the rest of the stuff in this list is derived from NIST guidelines

- Establish a mandatory security policy for your website/organization

- Make regular backups, and store them on a separate server; test your backups regularly (e.g. by restoring them to a staging/test server identical to production)

- Stream log entries (or routinely back up log files) to a separate server

- Don't use Windows

- Don't use (non-LTS) Ubuntu, Fedora, Arch, or any other Linux distro which prioritizes the "latest-and-greatest" features over stability; prefer more stable distros like (open)SUSE, RHEL/CentOS, Debian, or an LTS release of Ubuntu (though I'd personally wait a year or so after an LTS Ubuntu release or an openSUSE release to give Canonical/SUSE (respectively) time to shake out the inevitable bugs)

- Never install software unless it's absolutely necessary

- Never install software from untrusted sources

- Never allow the use of passwords for SSH logins; always use key-based authentication (or disable SSH entirely)

- Never try to roll your own authentication unless you really know what you're doing (or you really want to learn what it takes and have no interest in using it in production) and it's absolutely necessary; 99 times out of 100, there's a library for your language that already does it with a much more comprehensive threat model than the one you're envisioning (and in the remaining 1 in 100 cases, there are at least plenty of smaller libraries that handle individual components of an authentication system)

- If you do roll your own authentication, at least try to use something like OAuth via an existing identity provider so that you don't have to store credentials (and that so that you can lean on that provider's expertise)

- If you do roll your own authentication and your own credential storage, make sure all passwords are hashed (SHA2 at minimum) with per-password-unique salts

- If you do roll your own authentication, play it safe and force the use of HTTPS site-wide

- If you do roll your own authentication and your own credential storage, try to provide some kind of multi-factor authentication, preferably one compatible with Google Authenticator; no, security questions don't count

- If you do roll your own authentication and your own credential storage, allow and encourage users to use long passphrases

- If you do roll your own authentication and your own credential storage, don't force users to reset their passwords every $NUM days; only force a reset if there's reason to believe that the user's account is compromised

- If you do roll your own authentication and your own credential storage, don't require users to include certain character classes in their passwords

- Have it in writing that nobody working on behalf of your website will ever ask users for their passwords

- You can't have a data breach if you don't have any data; only store sensitive user data if you absolutely know for sure that you need it (hint: if it's for "analytics" or ads, then you don't need it)

- If you do need to store sensitive data, ensure that it is encrypted (most full-featured SQL databases support encryption at various levels of granularity; for NoSQL databases or non-full-featured SQL databases, you'll want to make sure data is only sent to the database in an encrypted state)

- If you do need to store sensitive data, play it safe and force the use of HTTPS site-wide

- Hell, might as well just force HTTPS no matter what; it ain't like TLS certs are expensive (thank you, Let's Encrypt!)

- Routinely audit things like GitHub/Bitbucket/GitLab/$SOURCE_CODE_HOSTING_SITE repositories, Slack channels, AWS/Heroku/$APPLICATION_HOSTING_PROVIDER accounts, etc. to ensure that only the people who should have access do have access

- Make sure all persons involved in developing and maintaining the site use long passphrases and multi-factor authentication wherever possible (most source code hosting sites and cloud providers offer 2FA, and quite a few - like AWS - can even enforce it on all your organization's accounts)

Wow this is a list aha. Thanks I favorite this question for future rerefence can't comment on the validity of your remarks (I probably don't know, though know some of it). Thanks

Edit: if you use encryption can you search it in real time like wildcard and ajax?

"if you use encryption can you search it in real time like wildcard and ajax?"

Depends on how it's implemented and what exactly is encrypted. Encrypted database fields have to be decrypted first; some databases (or extensions thereof) offer transparent encryption/decryption of fields, but others will require your search process to decrypt each record.

Generally, though, information that needs to be encrypted (read: anything personally-identifiable or otherwise confidential) really shouldn't rely on wildcard searching (I guess it depends on the use case, but this seems like it'd very easily result in information leaks). If a user has private information, it'd be more practical to just associate the encrypted data with the user's unique ID number and look it up that way.

These are great lists.

What about WAF (Web Application Firewalls)?

Reply on news.ycombinator.com