Ask HN: What are the most paranoid security measures for running a web app? | Hacker News Reader

Ask HN: What are the most paranoid security measures for running a web app?

4 pointsrobmclarty9 comments

I have a few different web app ideas and some of them I feel would require maximum security measures to be put in place (e.g., storing someone's medical history, or a system for tallying votes without corruption).

What kind of tools would you use for something that requires high security? Is SSL enough? Are there alternative secure transport protocols? Should the entire database contents be encrypted? How would that affect data integrity and backup strategies? Are there certain databases that are more secure than others?

Ideally I would want the web app to be designed so that anyone can connect to it with a regular browser without the need for a special client-side installation, but I want to ensure that the information being transmitted and stored is as safe as it can be.

View on HN

The first thing is to use full-disk encryption. Then, use ZFS because of its unique snapshot features. Use a battery-backed RAID to prevent corruption on power loss as good as possible.

Regularly back up your whole server disk off-site, and if possible, on tape as well as on disks. This way you can, if a hacker compromises your system, at least follow his tracks and minimize data loss.

Strictly firewall your servers - nothing except ports 80/443 public-facing; use a "bastion host" for SSH logins and only allow public-key based authentication (if you can, use two-factor auth, e.g. with SMS-PIN or Google's authenticator service).

For databases, use only database engines with atomic read/write and disable the write-back cache (i.e. force the database to only give a OK to the program when the DB change was provenly committed to disk!).

Always protect SSH and other keys with strong passphrases so that in the event your server gets compromised, at least a hacker can't easily decrypt your data. Encrypt sensitive stuff in the databases (credit card data, personal identity information).

Strictly limit management access to the servers to few employees. Protect the hardware with features like auto-powerdown if the rack/the server case gets opened. Disable the remote management interfaces present in many servers. Disable the firewire and USB ports in the BIOS of the server and protect the BIOS config with a strong password to prevent stuff like Firewire DMA memory access.

Have a disaster management plan written on paper, which details who is responsible for which operations part and how he can be contacted in an emergency. Use a second datacenter for hosting a fully-redundant hot spare (or production, if the code allows). Do not exclusively rely on Cloudflare for protecting your servers from DDoS attacks; CF is a prime target for failures.

Keep the ops documentation current. Document everything, because you can always be hit by a bus, arrested by cops or injured or whatever.

Great points. I've never used a bastion host before. Totally agree that documentation is critical.

Depending on your security needs, you can even go further - two linked bastion hosts. The public-facing bastion host has two NICs, one for Internet, the other one as a 1:1 link to the second bastion host (having two NICs again, the first is the endpoint to the 1st bastion host, the 2nd connects with the internal network).

Then make the public bastion host a VPN gateway so that the SSH bastion host can only be reached by SSH. This way, even if the first bastion host/VPN gateway appliance gets compromised, an attacker still needs a security flaw in the SSH bastion host.

Oh, and I forgot: regularly update your systems! Subscribe to various security mailing lists - this is the fastest way of learning about vulnerabilities.

And if you're running web apps: clearly separate the individual virtual hosts using either a full chroot solution or stuff like suexec/suphp from each other; same goes for the database users. Only grant necessary DB privileges (e.g. no need to give a webapp db user the GRANT capability in MySQL).

If you're dead paranoid like me, set up the application servers as VMWare (or other virtualization) servers, and for each webapp, create an own virtual machine. Then, link the VMs with an internal network and create a "bridge" server running nginx/apache/squid as reverse proxy (which can also serve as SSL terminator to avoid spreading HTTPS certificates over dozens of machines!). Maybe use special IDS software to detect/prevent common attacks like SQL injections. This way, if your web-app gets hacked, at least the attack won't easily spread to the other servers.

Thanks for all the advice! You've definitely introduced me to some things I've never considered before. Now to cram more learnings into my brain... :)

:D Yet another thing that just came to my mind: use different operating systems and software stacks on the two bastion hosts (or, at least, OS A on the bastion host and OS B on the "normal" hosts). You may, for example, choose a *BSD variant for the bastion host, and a Linux variant for the web hosts.

Simple reason: if the software stack of one of the hosts gets compromised (e.g. a 0day is found in OpenSSH), then the other bastion host cannot be compromised with the same exploit.

Use servers with built-in batteries (they're rare!) to allow erasing of the RAM contents in case of power loss or intruder alerts (e.g. if malicious entities break into the data centers and cut the power, the disk encryption keys in the RAM can not be extracted by e.g. spraying the RAM with cool-spray to preserve the content.

Set up issue-alerts for events (like: email all admins when someone opens a rack or a server case to prevent rogue employees from unnoticedly manipulating hardware). If you host your own data center: strict multi-level biometric access controls, require the presence of minimum two or three people to enter the secure data center.

Give the "doorman" of the data center a remote "kill switch" for the servers (this also can be a PIN code for the doors), so that if the doorman is forced to give entry e.g. by being hold at gunpoint or by a judicial order, the servers can be disabled without anyone being able to prove it.

Record and store video and every other logs regarding datacenter access to assist in post-mortem analysis. Take care to have at least two, better three or four, dedicated power supplies for the DC, not including the diesel backup engine and the UPS systems covering the time between power outage and diesel kick-in. Same goes for the data uplinks (fiber interconnections to other DCs). Guide them through different paths in the building and make the fibers/powerlines exit the building on different sides to prevent accidental dig-up by construction workers.

Regularly have your datacenter inspected by firefighting department, building safety bodies and every other regulatory body. This provides certification for possible other customers of your data center, as well as preventing closure of the datacenter for violating obscure legal requirements.

Have obscure passwords with keywords within them that are memorized in case your password file is stolen or cracked. Record and lockout failed login attempts (those records could be altered anyway.)

Check out PCI DSS regulations for storing credit card numbers, they're pretty strict.

Right. Will do.

Reply on news.ycombinator.com