How I spend my first 5 minutes on a server
plusbryan.com
plusbryan.com
1) Your first couple minutes on a server should be used to install a configuration management client, if your bootstrap policies somehow don't already install one.
2) Everything else listed in this document should be configured by a configuration management system.
3) "User account sync tools" should have no place in a modern infrastructure, you should use your configuration management tool to (at the bare minimum) deploy /etc/passwd and /etc/sudoers across your infrastructure.
4) You should not use shared/role accounts. The "incremental cost" is paid back immediately when someone leaves your organization; having to update everyone of a changed password or having a password change have any negative impact at all should not be a thing your company does.
This stuff isn't hard. It's worth doing right.
Can you provide an article as equally succinct as the OP's that provides this information? Your list is painfully devoid of anything of true value. Since it's not hard, and worth doing right, I imagine something should already be written.
I don't typically publish writings, but this seems like a good place to start. I'll write something up and post it here for the same critique that we've given Bryan :-)
In the meantime, a decent source of generalized (not succinct) Ops-type knowledge can be found here: http://www.opsschool.org/en/latest/
So you criticised the original article, but when asked to provide information or advice of your own merely came up with something entirely content free. The original article provided succint, useful advice, something you have failed to do.
This depends entirely upon your level of masochism and the kind of language youlike scripting in, as literally anything will do. Even shell scripts. Even Makefiles. Whatever you're most comfortable with, just start writing out configs. (you will eventually come to hate your job if you allow non-programmers to script/program in your CM, but blah blah keep shipping blah blah) Break it all out intoa wide hirearchy so you can reuse some bits in other bits.
Hey look, I wrote a crappy one! https://github.com/psypete/public-bin/tree/public-bin/src/si...
Next we implement the OP's comments.
Hey look, I already implemented it in my crappy CM tool! https://github.com/psypete/public-bin/tree/public-bin/src/si...
First push your CM tool and configs to the host:
scp -r simplecfm-0.2 remote-host:
ssh remote-host "cd simplecfm-0.2/ ; perl simplecfm examples/first-five-minutes/main.scfm"
But sometimes, and especially for servers that will be delivered to the customer afterwards, it's not practical to use a configuration management tool.
Also, millions of servers were deployed before Chef/Puppet appeared. You can't tell they did wrong.
Also, Chef/Puppet type solutions may be overkill for some tasks, fabric takes care of the easier cases for example.
Even a shell script that automates your standard install scripts is better than doing it by hand, because they can ensure you don't forget any steps and verify the state afterwards and ensure you don't forget any of the verification steps either.
You can put all his recommendations in a shell script (as the easiest solution) then run it (and if you ever did this more than ONE time you see the value in it)
Having an up-to-date system and only accepting security updates is a good policy. Fail2ban is a good tool (but it's a starting point; you should be doing other things to detect suspicious behavior).
The rest is just bad advice.
Having everyone log in using a single user account is a terrible idea. You can't audit who did what, ever. You have to remember to remove people from authorized_keys when they leave, and also make sure that they haven't left themselves a backdoor -- a cron job that reinstates the key, an extra user account, even just changing the root password to something else (how often do you actually check the root password on your boxes?).
User account management is a pain, so that's why we have things like LDAP. Everyone has their own user account. You can audit who does what on every machine, and for stuff that requires root, sudo will log the things people do (of course, if you let people have root shells, that's harder). The only people who get access to a local account (and/or root, but I still think root should just have a random password that no one knows) are a few sysadmins. When someone leaves, you kill their account in the LDAP server.
Even better, if this is a possibility, put up a VPN, and only allow ssh access via the VPN (using a firewall). Tie the VPN login to LDAP (and don't let non-VPN-admins ssh directly to the VPN server), and then you can be sure that without a user account in LDAP, no one can log into your servers.
Blind-updating systems in production is a terrible idea. Things break in the open source world all the time when you do this. Never ever use unattended-upgrades. You just need to be on top of security updates. Period. No excuses.
You should never even have a "my first five minutes on a server" type thing anyway. Rolling out a new server should be fully automatically operationalized. The first time you log into the server, it should be completely ready to go. It should be ready to go without you needing to log into it at all. This takes a small amount of up-front effort, and will pay off immediately when you bring up your second server.
Frankly, you need a lot of developers and servers before investing the time to setup a ldap deployment, integrating it with logins, and spending the inevitable hours debugging why nobody can access anything anymore, becomes more worth it than "just rsync/pssh into all servers and edit /etc/passwd".
Actually, if I had to do it, I'd use chef to automate creation (and destruction) of user accounts over an LDAP any day. Chef can be a pain to learn and use, but any sort of LDAP is even worse.
If you're dealing with more, let's say, 5, your suggestions became relevant
Also you are assuming this servers are staying with the company, which may not be the case
"User account management is a pain, so that's why we have things like LDAP"
Which is a bag of hurt in itself.
You're right -- many used cfengine. Still others used a custom 'config' rpm / deb that deployed all of these files everywhere.
Automated configuration makes sense not just for repeatability, but for auditability and documentation. Especially when you are going to 'hand the server over', the next admin should be able to know what you've changed.
Also, disallow password-based access to everything (use the keys, Luke.)
This line of thinking represents a logical fallacy - no one claimed that anything other than Chef or Puppet is "doing it wrong".
From the perspective of a sysadmin, this article has a lot of issues and it's inadvisable to follow its recommendations. Who doesn't use a hardware firewall? Who exposes ssh to the internet (requiring fail2ban) when a VPN server is much more secure and easier to use? Setting up an LDAP server is really easy and costs nothing. There's no excuse for shared accounts.
I, for one, trust ssh more than any other software wrt security, especially with password login disabled. Disclaimer: I am not a security expert.
Let's say that, hypothetically, a 0-day exploit was discovered in SSH which allowed remote code execution. A script kiddie begins trawling the internet for publicly accessible SSH servers to attack.
Your servers allow SSH from anywhere on the internet, and are eventually discovered and exploited. Mine, which will only allow SSH connections from my VPN bastion host, are effectively invisible to the attacker and will not get exploited (by this particular script kiddie, at least).
Adding a VPN server in front of SSH won't protect you from an APT, but it will protect you from 99% of the random, automated attacks that take place.
openssh is one of the most secure projects. It's developed by the security obsessed (and I mean it in a kind way) folks at OpenBSD.
I, for one, am ready to place for more trust in openssh than in any VPN daemon. The most commonly used ones are propitiatory.
What if there is a 0-day vuln (not exploit) for these VPN daemons? That far more likely. "Securing" ssh with a VPN is just one step beyond of security by obscurity.
If you are afraid of script-kiddies and scanners, let your sshd listen on a non-standard port.
So, just to see if I'm reading you right: you're using a VPN in the place of an SSH jump box, not making a judgement about the fitness or trust placed in your VPNd over your SSHd.
A reason to have separate accounts is that not only do you terminate access, you also have an easier time ensuring that less of what that person had access to could have been compromised. (This of course goes right out the window if said person has sudo/su access, in which case you have a much harder time, but even then giving them individual accounts means your opportunity to audit becomes so much greater)
After all, it's not the honest guy who'll never try to log in again you're primarily trying to protect against (in fact: for the honest people, a good security policy protects them by making them less likely to become potential suspects if/when something happens - it's in your own interest when you leave an organisation to ensure you get locked out), but the guy who might decide to try to do something later, or who might even be thinking about doing something before they leave.
I haven't, but if you assume actually malicious users you're probably going to end up with something so locked down it's useless. Aren't you?
Now, you must also have a functioning system, and so you may take risks by leaving things more open than you would like if you don't have the resources to thoroughly lock everything down.
But wherever locking things down further costs you very little, you should take the opportunity. And elsewhere you should asses what level of protection you can afford. Ultimately it is a cost-benefit analysis. Many risks are not worth spending money protecting against. Others are vital.
But even disregarding malicious users: Individual user accounts is not just a protection against malicious users, but against careless users. When someone sets a password that gets guessed, you want to be in a position where exploiting that persons credentials is as hard as possible, and tracking down actions taken via the account is as easy as possible.
And yes, you could insert something into a build script. But if the build script is committed, and the commit was pushed from a named, individual account, you're now at the risk of going to jail. Creating deterrents is often a sufficient risk mitigation strategy to be acceptable.
A developer is more likely to create better and more easily maintainable software if the target audience is assumed to be an ordinary user with no special system privileges. In my experience, when a developer has root and assumes everyone else does, deployment becomes a nightmare.
What I was trying to say was that there's not really any way for you (server admin guy) to know if I (software dev guy) have inserted something malicious into a script that all the other software folks run constantly (software build system, NOT server build/init script, NOT deployment script).
This is not about the end-user's privileges, or server set up, just how in a team-base software dev environment you're probably going to have to have a measure of trust for your employees.
They are not security tools. So you're on your own on what to actually tell the tools to do. "Install chef" is not a security tip. It's a repeatability tip, so you can get your system up to a known state repeatedly.
For the security side of things, you're back to figuring out what the right steps are, no matter how they're installed.
Secondly, you are about 4-5 hours away from learning puppet (or Chef) and making this checklist into actual code.
Thirdly, you now have a checklist of items that you can use in a job interview if you get the oppertunity to gain a new-hire or an intern.
Lastly, good on you for submitting this to a peer-review on HN. We can be a picky lot.
TL;DR Checklists are a good first step for building a proper config management system.
For what it's worth, some of the advice given in that article was worth mentioning (eg fail2ban, it's a great tool). But the shared account suggestion was the complete opposite of how you should be managing user accounts as you lose audit trails. And for that comment alone, I'd recommend people read that article with a degree of scepticism before rushing onto any boxes they might administrate.
That article has generated a lot of good discussion though. So even if just indirectly, it's been a valuable contribution to HN.
> I'd recommend people read that article with a degree of scepticism before rushing onto any boxes they might administrate.
Agreed. But the same could be said of everything; hackers and engineers, empiricists both.
Some of the advice in the article would seem apporpriate to someone who had never worked with another senior engineer. No fault of the OP.
System engineering and security are as much a learned craft as a science; a dialectic between sand castles (if you will).
vim /etc/sudoers
The ufw man page is pretty decent.
For anyone else that followed the thread to this point- this advice on bringing iptables back up on reboot worked for me http://rackerhacker.com/2009/11/16/automatically-loading-ipt... YMMV
The advice is not complete. IPv6 is real and really works most of the time these days. Back up your ip6tables to a file too. I like /etc/firewall-4.conf and /etc/firewall-6.conf but it's down to preference.
Know about iptables-apply too, lest you be caught unaware.
EDITOR=emacs visudo
%admin ALL=(ALL) ALL
So you just need to make sure your 'deploy' user account belongs to the admin group and you're good to go.
usermod -a -G sudo username$ sudo adduser username sudo
2. If you're on ubuntu, root already has no password, and your initial setup user (whether it is called "deploy" or "kilroy") is in the sudoers file.
3. Other things I install in the "5 minutes with server" are: htop molly-guard screen git-core etckeeper
git-core because I prefer my etckeeper in git, but if you want it in bzr you don't need git-core. INSTALL AND CONFIGURE ETCKEEPER AS SOON AS YOU CAN, seriously. You need it. You'll thank me when you try to figure out when and how something in /etc got borked. (you need to edit /etc/etckeeper/etckeeper.conf if you use gif. You need to do "etckeeper init" and then "etckeeper commit" to establish the baseline)
molly-guard stops you from rebooting the wrong server
screen (or alternatively tmux) lets you keep your session open through ssh session disconnects (e.g. when moving from wifi to 3G, or between 3G towers that give you different external IP). The most useful way to use screen is "screen -xR" which also lets you share your session with someone else should you need to.
The server should also be rebooted. Applying kernel updates makes no good if you never apply them!
screen -x is equivalent to screen -xR afaict:)
I'd also add @reboot screen to crontab, which will recreate a session on startup - in my bashrc I have :
if [ "a$STY" == "a" ]; then screen -x fi
Other useful things include actually setting up backups (duplicity is a useful first step here), installing munin/nagios to monitor the new box.
Realistically if you are doing this more than once per blue moon, then you should be using something like puppet to do this automagically.
"screen -x" requires a session to already exist, whereas "screen -xR" will join one if it already exists, but will create one if it does not. At least it does in v4.00 which I use. If you have a "screen" call in your @reboot, you already have a session, so they will work the same.
etckeeper is a net that gives you an idea of how /etc looked on a given date or apt-run, not just how it was supposed to look (Which is what you get from the chef/puppet logs).
Question to chef/puppet users: do you restore the state continuously? or only when you've made a change?
I have no experience with chef/puppet, so I might be mistaken, but I'm under the impression that they only push changes when asked to - which means a local change may survive for several weeks before it is overwritten. But I could be wrong about that.
You seriously do this by hand for every server? That seems error prone and a huge waste of time when tools like puppet and chef exist.
As an aside, I think the default fail2ban config is too loose and quiet. Here's [4] an Ansible task file that configures it to be more aggressive and send notification emails.
[2] https://gist.github.com/dbarlett/5079802
sudo("apt-get install -y <name your packages>")
- name: Install prerequisites for PPA management
apt: pkg=$item state=present update_cache=yes
with_items:
- python-software-properties
- software-properties-common
I'm not trying to be provocative or negative. I'm really trying to understand the supposedly big difference between what's labelled a deployment tool, and configuration management tools.
Deployment tools just execute whatever script you hand them, and so the scripts are either more brittle (server must be in a precise state beforehand or it doesn't work right), or require more effort to duplicate the work that the configuration management software does to only make the needed changes.
If your deployment scripts get complex, it's more difficult to see at a glance what the end configuration is supposed to be.
Your example is great to illustrate why configuration management is so powerful.
Most importantly, your example will upgrade the package to the latest version every time you run it (which is probably not what you intended, but maybe it is). I'm sure there is an idempotent command for each of apt, yum, ports, homebrew, etc, but I have better things to do with my time than figure that all out.
In puppet your command would be:
package { 'foo': ensure => installed } # or 'latest'
For RedHat systems, it would be:
package {'foo' : ensure => installed }
package {'foo' : ensure => installed }
package {'foo' : ensure => installed }
What if you want to build an haproxy configuration file that pulls the list of hosts from an authoritative source? Do you "just" build it up from scratch every time and replace the file on every run of your script? With chef/puppet, you can pull that information in from your node classifier using the same query for dev, staging, and production and have the same recipe work for multiple environments. In some environments, the haproxy config has now changed, so I want to tell haproxy to reload. But in other environments, the config stayed the same, so I don't want it to reload. The file and service don't get touched if they didn't change.
What if a service has multiple files associated with it that when one or multiple of them change, you have to restart or reload the service? You don't want to restart each service every time one of the multiple files changes, so you end up having to build a queue to handle the service notifications.
Say I've got a few dozen users on my hosts and someone leaves. Which is easier - writing a purge script which knows how to purge the user from all environments that you support, or this...
user { 'joe' :
...
ensure => absent
...
}
multiple packages can be done this way if you have a lot:
$prep_packages = ['package1', 'package2', 'blahblah']
package { $prep_packages: ensure => installed }
You can of course use extra arguments if you need them.
We're currently using local runs of puppet, and it's easy to do dry-runs and see what fails before applying when testing. I'm not sure how Ansible gives feedback on that (I assume it must) but I find puppet pretty good about it.
You can bootstrap 0mq and run them over that instead. That's much faster.
Speaking from experience as a part time admin, it is definitely worth it to automate the process, even with one or two servers. Any build system will repay in spades the first time you have to set up a new server or rebuild an old one, esp. if you're in a hurry after a server failure.
That said Linode offers other easier options (Linode specific of course). You can take the commands usually executed, and put them in a shell script in the language of your choice which is uploaded and executed on first run (they call this StackScripts - there are lots of examples and helper scripts on their site, but the scripts are really very basic). It's perhaps easier than learning a DSL as it's just codifying the same commands you would run via ssh by hand. You can also clone nodes so you could set one up in a known good state and clone that each time.
This does tie you to linode, so long term one of the other options available would be worth looking in to.
Comments like those are why I normally point people to actual security expects (like, say, Schneier), and why I recommend that new admins should ignore as much as possible the practices chanted by the industry. A secure server does not need a firewall. A firewall can be used to secure a server against a specific threat, but that's it. The days of ping of death are behind us.
I would like to point out that following the article's guide and firewalling away ICMP, you can end up with a lot of trouble. (see http://serverfault.com/questions/84963/why-not-block-icmp). Some ICMP messages are not blocked by default by ufw, so I'm unsure how damaging ufw is when used like this.
At any rate, a Firewall is a block. A new fresh server install won't have ports that needs to be blocked. By putting up a firewall, there is nothing to be gained. Before the firewall, the ports are closed. After adding the firewall, the ports are closed. All that is gained is a hurdle the next time one wants to install something like a monitor tool (like Munin), or a new service.
It might be useful as a last line of defense against malware regarding outgoing traffic. I am normally against that kind of thing however (as focusing on the cause is better than the effect). At best, one can catch a spam malware, but any bot net, web server, ddos or other type of malware are untouched by the rules (port 80 and 443 is allowed). If the server has email sending configured so root message can be sent, then the spam malware can use that route and the firewall will just sit there.
So let's take a newly installed machine. What threats can be identified and what risks are we trying to mitigate with the help of this firewall (as specified by the article)? The only thing I can think of is either a Zero day TCP/IP stack vulnerability (not a realistic threat), or that the admin doesn't trust the other admins when they install new services. Yes, if an admin installs a new email server and enables relaying to the whole world against the explicit recommendation in bold font by the install wizard and the configuration file, a firewall can block that admins' actions. Then again, that same admin could just as well have disabled the firewall to "get the mail to work", so I'm not sure it's a viable defense against bad admins.
You are correct that a firewall will not magically solve all your problems, but it does help to protect against programs that open ports you didn't know about.
Recommending against them doesn't make sense, and implying that they are only useful to prevent TCP/IP zero day vulnerabilities is silly (especially since the firewall likely wouldn't protect against that anyway).
This is about as far from a server installed with ubuntu in 2012 that one can get. You are not going to find any such article by Schneier promoting default firewall installations. I suggest here to check out Secrets and Lies by Schneier, as it is rather clear that a firewall need to be configured against the specific threats one can identify. If you fail at identifying threats, the firewall is likely not be useful at all, or will simply work identical to NAT. At worst, it will give a sense of false security.
I think it's less about defense against "bad admins" than it is about protecting against accidental bone-headedness. :-) I typically set up a restrictive firewall policy even when I have a clear list of the services I'm running and/or I am the only admin. This comes in handy every once in a while, in cases where...
* A service is expecting more ports to be open than are documented. (Happens not-infrequently with license servers.)
* I'm re-using an old image and there are undocumented services enabled by default.
* A user decides to run a network service in their own account without informing the admins.
In all those cases, am I likely to change the firewall to "make it work"? Sure. But having to actually make that change helps keep an audit trail, and helps keep the admins explicitly aware of the attack surface. It's similar to why it's a good idea to periodically run nmap against your own servers.
Here nmap do shine, and periodically running nmap is a technique that should be taught in universities. Great way for students to both learn about computer systems, and about learning how to debug problems.
The days of ping of death are behind us.
Flashback quite a few years. I was working in IT and my coworker asked me if my WinXP (IIRC) machine was up to date. I said "yes". Next thing I know, it crashed hard. Oops, my buddy just hit me with a ping of death.
If the FW itself has that specific NIC, it could be brought down with this attack - but you could prevent SIP traffic from hitting machines which are vulnerable.
Robust, redundant deployments should be a part of an overall security policy, securing yourself from outages and downtime.
It's also a good idea to ensure that sshd has fresh keys that are unique to that machine. Hopefully, your images are installed without sshd keys, otherwise you'll have multiple servers with the exact same keys, which is considered bad practice. During initial configuration before deployment, you might want to remove the keys so that sshd will create fresh ones when it starts:
rm -rf /etc/ssh/ssh_host_*I had fun this past November with Snori74's Grep101 course.
I don't think it's open right now, but when it runs, it's a great stab at exactly what you're describing.
Initial blog: http://grep101.blogspot.com/
Cool trick: use ssh-copy-id <server> from the client machine.
From the man page:
ssh-copy-id - install your public key in a remote machine's authorized_keys
Also a big fan of externally verifying what ports are open, and making sure the system is in monitoring, backup, config management systems. Config management is kind of optional if you have a small number of servers which don't duplicate configurations, though, and there's often no need to back up the OS, but any data should be backed up automatically.
netstat -ntap | less
ps aux | less
Check for unusual daemons, ssh running on other ports (yes, the provider pre-loaded systems with a back-door ssh without disclosing it to us).
This is especially important when you are taking over admin on a server you didn't setup yourself. Other folks have weird ideas on how to admin things. Like webmin for example...
I also like epylog for finding unexpected stuff in the logs.
what's good beyond this:
chkconfig
cat /etc/rc.localGenerically (for Linux), take a look at /etc/init.d , /etc/inittab, or /etc/rc?.d.
It'd sure be nice for those of us who are not security experts to read alternative approaches rather than, paraphrasing and not picking on anyone, "using a firewall is dumb" or "blocking ssh is pointless".
I like isolated ideas such as using a script to completely automate the provisioning of new boxes. Kind of a no-brainer if you ask me. The problem is that such recommendations are not followed by something like "Here's the script I use on Ubuntu 12.04 LTS".
How about it guys? Would you care to attempt to produce a canonical HN "How to harden your server" reference?
Maybe one of the security experts on HN can start a repository on Github to evolve a canonical script. I'm pretty much 100% Ubuntu 12.04 LTS, so it is my hope that this is one of the platforms that is addressed.
I did some looking around and this is what I found (I am in no position to evaluate the merits of any of these at anything beyond an intermediate level):
https://github.com/bluedragonz/server-shield
https://github.com/eglimi/linux_hardening
http://www.cyberciti.biz/tips/linux-security.html
http://ubuntuforums.org/showthread.php?t=1002167
http://www.thefanclub.co.za/how-to/how-secure-ubuntu-1204-lt...
http://www.andrewault.net/2010/05/17/securing-an-ubuntu-serv...
http://ubuntuforums.org/showthread.php?t=1919111
https://help.ubuntu.com/12.04/serverguide/security.html
http://www.sans.org/score/checklists/linuxchecklist.pdf
http://nvd.nist.gov/scap/content/stylesheet/scap-rhel5-docum...
http://blogs.csoonline.com/ubuntu_lts_vulnerability_scrub_ag...
#1: A good password is a must. During installation, have a second computer generate a good password and either memorize it, or GPG encrypt it somewhere on the second computer. pwgen is decent in generating passwords.
2#, I fully agree with the article on automatic updates if its a personal computer. For others, one can have root mails sent if you are a fast and and read mail daily. This is how many people read about vulnerabilities before they reach the news.
#3: When installing large package like web services, I keep in mind of the long term prospect of each project. I ask the questions: Is there a deb package? Is it being maintained by a large group of independent developers? Is it mentioned in discussion at Serverfault? Are there any recent updates? What does the Wikipedia page have to say?
#4: read the man page, and check any section labeled security. Some man pages will say things like "we have this port open. Its completely insecure, and we expect either the local network to be safe or that you use a firewall". Through this just happened once for me, its still a good practice to check the man page with new services.
#5, avoid php themes/mods that require you to manually patch things. They won't be updated by Ubuntu, so things will either end with you uninstalling it or forgetting that it exist and thus get hacked. Sometimes ubuntu will just install over the mod, dealing with the issue for you.
Other than that, harderning depend on use case. A wiki/forum will need some form of anti-spam protection. A media center need access control or firewall to only allow local network. Unsecured protocols like nfs and nis need something like ipsec. A mail server needs authenticated smtp.
Why a second computer?
http://practicalops.com/my-first-5-minutes-on-a-server.html
Happy to answer questions about it. :)
Towards the end of your page you talk about keeping a manual log file. That's exactly the way I've handled this sort of thing for years. I usually call mine "project-log.txt". These files usually have several sections and it looks like this:
Header: Project name, start date and other relevant details
Date entry: What I did on a particular date
Working on now: Before I do anything I make this entry. After an interruption I usually have no clue what I was working on. By making a habit of writing a note to myself about what I am going to be working on immediately I can task-switch to that phone call or unwelcome question and mentally come back to what I was doing quickly.
To Do: Self explanatory.
Questions: If questions come up I log them here so I don't forget.
Ideas: Sometimes as you are working you say things like "hey I should do x". I log them here and go back to what I was doing.
References: Links, etc.
After launch: There are things that might not be critical at all and can be done after launch. I try to only note must-have's in "To Do".
This file lives at a self-named directory: "~/project-log". I do this because I can also save other relevant files there and keep the entire thing under source control.
<Project Name>
<Date>
<Description>
<Notes>
--------------------------------------------------------------------- 01MAR13
<stuff I did on this date> format example:
- Installed Ubuntu 12.04 LTS
- Installed PHP
- Installed MySQL
- Installed Python
--------------------------------------------------------------------- 02MAR13
<stuff I did on this date>
--------------------------------------------------------------------- 04MAR13
<stuff I did on this date>
--------------------------------------------------------------------- WORKING ON NOW
- Configuring Virtual Hosting
--------------------------------------------------------------------- TO DO
- Configure vim for php development
- Customize bash
- Install imagemagick
- Install php5-imagick
--------------------------------------------------------------------- QUESTIONS
- How to automate server bring-up and hardening?
- Firewall or not?
- Disable SSH?
--------------------------------------------------------------------- IDEAS
- Learn Ansible for automated deployment.
--------------------------------------------------------------------- REFERENCES
- Ubuntu 12.04 LTS docs
https://help.ubuntu.com/12.04/
- Ansible
http://ansible.cc/Nowadays I'm downright spoiled and use org-mode[1] to keep my systems journals. Org files are plain text as well, and org-mode takes care of setting up the tree by date. I can also add a journal entry from anywhere in Emacs with just a couple keystrokes, which makes it incredibly low-friction to use.
Like I said, the most important thing is to TAKE NOTES. Even pen and paper. It's one of Limoncelli's big points in Time Management for System Administrators.
Tooling doesn't really matter, the important part is being able to remember what the heck I did and when I did it. Invaluable for troubleshooting.
[1]: http://orgmode.org
Most extensive security procedures contain a lot of questionable advice and few will prevent human error which most compromise can be traced back to.
The only semi-universal list amounts to 1) use keys for ssh
2) block/turn off everything except your (web) service
3) automated security updates or some sort of failsafe procedure to install security updates very regularly (this is by far the most common error)
4) Learn about web application security threats especially sql injection and consider them at design and implementation and have some kind of regular review (this can be very difficult to get right)
5) Avoid storing anything that makes you a particularly desirable target like bitcoins or secret defense plans as anything can be hacked with enough incentive.
Most everything else probably won't make too much of a difference to you.
http://www.nsa.gov/ia/_files/os/redhat/NSA_RHEL_5_GUIDE_v4.2...
and TLDR version: http://www.nsa.gov/ia/_files/factsheets/rhel5-pamphlet-i731....
It is starting to get a little bit dated (RHEL 5 is quite old), but general rules still apply and usually they explain their reasoning.
This isn't clear. Are you saying you think automating these updates is good or bad?
> 5) Avoid storing anything that makes you a particularly desirable
Well, in some cases your user's uid and pwd is the most valuable chunk-o-data a would be attacker wants.
> Most extensive security procedures contain a lot of questionable advice
List?
> and few will prevent human error which most compromise can be traced back to.
I think this would be the power of having a canonical auto-provision script on Github that many can review and contribute to. The script could certainly take the form of sections that could be commented out as needed. In other words, a well documented and reviewed set of recommendations that someone could edit based on pier-reviewed information in the comments and then use to automatically configure a server. That, I think, could be of value.
It seems that using puppet has a pretty steep learning curve, at least from what I could gather skimming the docs.
Actually what I would really like to see is an interactive script, which then guides me through the process of hardening a fresh ubuntu server install, offering sane suggestions along the way.
I am pretty sure such scripts do exist, but having them maintained and regualarly updated somewhere would be quite neat.
Today I have started with ansible and have already gotten a huge amount done and deployed. Very happy with it. It's all in a small git repo and its very readable and concise.
It would offer you explanations, deeper explanations, defaults, recommendations for when you would override the defaults, etc. for each item. It would be smart enough to prevent incompatible or contradictory settings.
Once this process was complete, it would generate the needed puppet / ansible / whatever script, which the admin would run and store for future reference and use. When installing another machine, the original script could be read in by the decision tree and the new script could be generated by modifying the original rather than starting from scratch.
If you are walking past a dark alley in a big city, you don't say, "I am a blue-belt in karate, and I have a wallet with $300 in it." They have a gun, and they'll take that wallet, thank you.
I totally appreciate you putting all those links together, but I would hope that others remember that loose lips sink ships when it comes to security. And from a hacker's perspective, what's the fun in social engineering if someone just blabs it all in a post?
Real would-be intruders are not dummies. They have a suite of tests they can run to "x-ray" your system to the extent it is possible and discover vulnerabilities.
To some degree it's like encryption code. The safest code has to be open source.
This is a commonly held belief that is not true. I've been doing open source development quite a bit over the last several years and have seen plenty of insecure open source projects that were even less secure than I would see in the private repository of every place I worked. Here's why:
* Open source code gets more eyes on it, when it is well-used. But there are loads and loads of projects that are hardly looked at, and they have a greater chance to be used before they are thoroughly vetted.
* Those projects were most likely thrown up there by a developer like me who just hacked something up quickly to solve a problem. Once done with the problem, the code stays up there and I just let it atrophy. That consists of 95% of my projects, at least.
w
last
dmesgI'm not sure it helped, but it made me feel better.
Sysadmins with root access should be able to handle a random 8 character or longer password, and that number is large enough for a secure public accessible ssh. If your not a sysadmin or unable to remember a random password, try go with a passphrase like "correct horse battery staple". If you have too many machines and thus can't remember passwords, then use a master password locally with a bit larger password and store certs there.
To do some math to illustrate the security of a random 8 character long password, someone would need to fill a 100/100 line for several hundred years to get through them all. By that time you should have noticed the constantly full connection, and be happy you haven't died of old age yet even after your 200th birthday.
However, 8 character passwords are not the suggested length by security experts. They suggest using a 10 character long password, as that is also secure in the case that your password hash somehow got lost.
For the several servers that I have, I have been more worried about the logs from failed attempts than I have been of anyone guessing the password. Logs wears on the hard drive, so one might want to install fail2ban to lower the number of writes to it. It also decrease the noise level in the server room.
Disabling remote root login,
1. cheaply reduces attack surface
2. necessitates assigning administration rights to specific persons and roles.
3. greatly increases command visibility and value of audit trails.
4. combines extremely well with disabled terminal shell over SSH
5. can reduce (unnecessary) system resource usage.
Etc.
There are very good reasons why this practice is standard. Unless you are specifically creating a test or honeypot box of some kind, you would be foolhardy to ignore it.
Control is an ignorant tyrants last redoubt. Security is only about: identifying risks and dealing with those risks. All other messures are worthless as actual security objectives.
I am not arguing with you in priciple, only in particular. All of your points are well taken, to reduce risk, but not to control.
The problem I have with the articles claim is that limiting roots access in ssh is not improved security. It does not acknowledge the threat or attack avenue at all, but rather tries a blind approach. The threat is script kiddies. The attack avenue is bad passwords combined with password authenticated use of ssh by root. One can then either fix A) the password, or B), the access root has in using ssh. I pick option A and thus have have good passwords that secure even against internal users, rather than having bad passwords and limiting roots use of ssh. Having good passwords, and disable ssh for root is redundant and a expensive limitation for no security benefit. Having bad passwords and disabled ssh is bad security practice, as bad passwords should by policy be impossible.
> 2. necessitates assigning administration rights to specific persons and roles.
If you need to have different assigned administration roles for a single installation, and guarantee a correct audit trail, you should look into physical tokens or at least a kerberos+ldap setup. "root" user should then not exist as something that anyone should use. however, if you do not care about guaranteed audit trail (and to be honest here, the common use case does not require kerbros+ldap setup), you can add something like environment variables to indicate which user is logged, and ask administrators to document changes to the system.
> 5. can reduce (unnecessary) system resource usage.
This goal can be mitigated in several way. Thus in a honest discussion, this should be the target of it and not the pretense of increased security.
If you know the difference between that and root, please explain it, because I can't see any.
It also becomes easier to detect when someone is attempting to break in when you can see logs of common user names in a row fail to log in.
Disabling root also means you now have to guess the username and the password instead of just the password.
Commonly, I track access through environment variables (configuration on the ssh client), and documentation that each administrator do after finishing a task. As one should not recruit administrators that do rm -rf /, tracking (blame) is of an lesser importance and documentation a much higher one. If more was required, I would go with kerberos+ldap approach without anyone using local root user accounts at all.
By their SSH-key and IP address (both of which are logged to a remote syslog-server).
Given that you should be discouraging root use and using sudo or scripts to automate admin tasks, why are you encouraging admins to log on as root?
That is a false premise. I'd never encourage anyone to use "sudo" because it's a source of errors; people get the escaping and context wrong all the time, doing things that they didn't intend, and wasting precious productivity on getting the incantation just right.
It also doesn't add any meaningful security, it's a red herring.
If you need a dependable audit-trail then sudo is worth zilch. In that case you're looking at SELinux, auditd and friends.
If you don't need that trail (which is most people) then sudo is a ball on a chain and a false sense of security. Local privilege escalation exploits are dime a dozen, any shell-account can be upgraded to root by a dedicated attacker anyway (sometimes via sudo itself, as seen in in the recent sudo exploit...).
Oh, and let's not forget: If you enable a user to do anything meaningful with sudo then that normally includes giving him a way to write arbitrary files as root...
I think you're doing something wrong with sudo. I'm not talking about "reentering your password" in the Ubuntu or Mac sense. I'm talking about giving access to specific limited commands.
And what distro are you running where local privilege escalation is common?
By definition the set is either so limited as to be essentially useless, or it opens (usually multiple) straightforward paths to a root-shell. Pretty much all sudoers-files I have seen fall in the latter category ("but he can only write the apache config-file!").
And what distro are you running where local privilege escalation is common?
https://www.google.com/search?q=linux+local+privilege+escala...
Or change the ssh port.
If you've got a physical safe that you keep money or valuables in, do you put it out at the kerbside so people can have a go at cracking it, or do you hide it inside in an obscured location to make it more difficult to have a go?
Surely it's the same with services on your servers, why needlessly advertise their presence. It's a tiny veneer of security but it is still more secure surely.
Also, since this post is for beginners, you should mention about restarting the sshd over the lish tty and not over the ssh ptty.
2. And you're not using a configuration management tool (like SaltStack, which is also a remote execution engine) this will give you: - a central point to manage all your server - predictable configuration on all servers with the same role - a configuration documentation place (and even history if you git the confs) - will make managing multiple users a breeze
3. Use VPN and private services on private IP.
Just as an example, the shared user account with unique SSH keys per user. Sure, it's obnoxious in some respects, but many of the criticisms I'm reading in the comments like "but they could reinstate their access with a cron job that re-adds their key when they leave" and such are silly - presumably those who are using the shared account are developers/sysadmins with sudo privileges. Regardless of whether they have a shared account they have privileges to do whatever they feel like to the systems in question. Hence I'd argue it's a fairly reasonable solution for the situation when you don't have the time/resources to configure something more complex and you have to trust all parties anyway.
I think there are two larger takeaways:
First - Managing multiple users across many servers and dev systems is not easy enough, particularly for smaller organizations, and only gets worse when you try to get more granular about who can do what.
Second - umm... no idea anymore. Forgot what was second. Automation is good?
chown deploy:deploy /home/deploy -Rfflag arguments should go before any other arguments to be compatible with the most Unix systems.
Ossec provides (among some other things):
* Log file monitoring, with severity levels, occurence counting, time-based rules and auto-response. This means (for instance) you get to watch your auth.log for failed login attempts and after 10 failures fire a script to ban him, alert sysop by email or have hubot alert your sysops. Or whatever floats your boat.
* File integrity monitoring. Make sure noone's been mucking around with your files. It has real-time support (through inotify), but if you don't want use that, make sure you store the database it keeps off-server for forensics if need be. Pro-tip: FIM and auto-updates are a tad unnerving.
* It can watch command output. You can use that to make sure the `netstat -tnap` output doesn't change, for instance.
* For larger/more compliant instances, it has a client<->server setup available.
- heavyweight;
- not actively developed.
Are my perceptions right? If no, how would you recommend to start using it? Any good tutorials or other pointers?
It doesn't feel heavyweight to me. It does start a bunch of daemons for all of its processing and it has the client->server bit built right in. That may make it feel heavyweight, I guess. But, you don't have to use the client->server stuff if you don't want to. It'll still do all of its magic for you.
Ossec will do a lot out of the box. So, I suggest installing it with the default ruleset and the active-response stuff turned off (the installer will ask you). Then dive into the rules and ossec.conf (knowledge of regex is required).
Documentation @ http://www.ossec.net/doc/index.html
I think more than anything it is just kind of at a near end-state where there isn't a whole lot left to ADD other than some signature type updates, or nice-to-have features.
It is a program that I HIGHLY recommend having and it is very simple to get up and running, even if you don't have packages of it for your OS version. Also you won't pass PCI-1 without a IDS like this.
Since the author is using ufw to control iptables, better to just use "ufw limit" rules for SSH port 22 to slow down the rate of any automated SSH bots trying to give your server a workout.
The chances of both that and the deploy user getting corrupted at the same time is unlikely.
PasswordAuthentication no
Honestly, being a private key screw-up away from never being able to log in again scares me a little.
So to get around this you either have to allow SSH from anywhere or you have to use some remote KVM system. Most of the remote KVM systems seem to be based on Java applets which is not really something you want to enable on your system.
So what is the best way to get around this? Just open up SSH and be diligent about your SSH security? implement port knocking?
Set up a small AWS server to use as a Web proxy and SSH proxy.
Connect from you home with a remote forward port (for example 2222 to 22) additional to any other local forward ports. Put this tunnel in the startup scripts.
When you are away, connect to your AWS server, then connect locally to port 2222.
Now you've got SSH access to your home machine from anywhere.
I'm just thinking out loud, I've never tried any of that!
I thought of setting up a VPN on a server, but all that would really do is move the problem from securing SSH to securing VPN.
One correction for you: the sshd_config lines should have no "=" symbol.
I just use this bash script when setting up a new VPS. It pretty much takes care of it all for you, plus it sets up a ngnix/mysql stack already optimised for law RAM machines.
Also sets up 3proxy which is handy for viewing Hulu/Netflix :)
Now imagine the conclusion after reading this thread. As some one who can just about get something useful done in Linux, this thread makes me want to never use it again, it just looks too scary. Loads of disagreements which seems to have lots of dire consequences. OK, great discussion for deep geekery, but scary as hell for normals.
Now, when ever I see those annoying posts from smug Linux users who jump in every time a Win or Mac user highlights a problem, I now have this discussion to point to as to why Mac and Win users wont generally go near Linux.
Sorry chaps (and chapesses), I am on your side, Linux is a great thing, but this has to be the worst advert for Linux ever.
The rest are decisions that you don't need to rush into, and things you should learn when they make sense. Be advised that "the proper way to configure a server" is an elusive beast, you'll be always improving it, it's never done.
Sure, you can put Ubuntu in your mom's laptop and she'll be up and running in a breeze, but hey, she's won't be reading Hacker News.
Does anyone have good reusable Puppet (or bash scripts) published that they use on all new servers?
* Puppet, Chef, Ansible, Salt, CFEngine, etc.
Looks like an interesting piece of software nonetheless.
> The days of passwords are over. You’ll enhance security and ease of use in one fell swoop by ditching those passwords and employing public key authentication for your user accounts.
ssh keys are better than passwords only because they contain (and require) more information. On the other hand, if your dev's machine is lost or stolen or compromised, so is your ssh key. This is especially a problem in environments with a shared account with full access, as you have. So, it's probably a good idea to make sure you're using a passphrase with your ssh key (during ssh-keygen), unless you need a passwordless login for a shell script or other automated remote system.
> passwd deploy: Set a complex password - you can either store it somewhere secure or make it something memorable to the team. This is the password you'll use to sudo.
Not necessarily. Anybody with access to the "deploy" account can use "passwd" to change its password to anything they like. (Edit: I'm wrong on this! passwd does require your current password; I've just gotten used to doing it for other accounts via sudo, which doesn't.) Changing the passwd on your own account doesn't require sudo. For this reason, I think it's better to simply give deploy nopasswd access to everything, and then delete and lock deploy's password to prevent it from being used at all (passwd -d -l deploy). You'll have effectively the same amount of security, but this way nobody will need to remember or retrieve a complex password, and you'll prevent, say, some accident in /etc/ssh/sshd_config from making deploy remotely accessible via a password.
You can do something better than this though, but it takes a little effort. Deployment is often the same steps over and over again (an rsync or an occasional apachectl graceful in my case). You can give the deploy user nopasswd access to only a shell script that's writable only by root; this way, deploy can still do 90% or more of their job without ever being given system administrator rights. You do have to be a little careful writing shell scripts though -- $* and "$@" still trip me up once in a while.
> Logwatch is a daemon that monitors your logs and emails them to you. This is useful for tracking and detecting intrusion.
This seems of dubious security value to me -- probably better as a generic sysadmin tool, so that you get annoyed by noisy logs and seek out and fix minor problems instead of ignoring them. Thing is, if someone does get access to your server, you pretty much can't trust it at all anymore. With services like Linode, you're really better off just launching a clean new instance, re-running your setup script (if you have one), and moving your data over.
I had to deal with the occasional intrusion in some pretty icky servers at an ISP once upon a time. We used rkhunter for a while, but I learned pretty quick that successful attacks against Linux servers are plenty enough sophisticated to alter all the basic tools that you would use to detect and remove the rootkit.
There is one caveat: I've been playing around with the idea of setting up rsyslogd to route syslog messages to mysql, and then using mysql replication to have an up-to-the-second offsite copy. I'd combine that with Snoopy (https://github.com/a2o/snoopy) or something similar. The point isn't to try to clean up an intrusion, it's to see how the intrusion happened so that I could close that hole. I haven't gotten around to setting this up yet, so I can't say anything terribly smart about it.
Finally: if you're going to have a problem with unauthorized access to your Linux or BSD server, it's probably going to be via one of its services, not via brute force ssh or anything similar. So, if you're concerned about this kind of stuff (and if you're being paid to be a sysadmin, you have to be), then you need to spend most of your attention making sure that your various services are set up correctly (apache/php/mysql/postfix/dovecot/spamassassin/etc. in my case).
The big problem with ssh keys is not being able to enforce ssh key passphrases on users. From the server perspective, you have no idea if the user has set up a passphrase. There are security standards which mandate certain kinds of passwords (complexity) and are silent on asymmetric keys, so you couldn't use keys in those environments.
The old solution was to do some post-login hack to require a password as well (e.g. to su), or do a VPN (which could have multiple forms of auth) and then ssh with keys after that, but the newest ssh (and I believe commercial ssh for a long time) now supports requiring multiple authentications per login, so you can do ssh key plus passphrase.
There are also DLP/etc. reasons why ssh can be problematic in some environments (i.e. where you're required to log/analyze actions taken by users, particularly admin users). The solution there is to use a bastion host and ssh in and then ssh out, with the user account locked down to log. SSH Communications (the commercial ssh people) have an interesting ssh MITM box which essentially does what all the SSL x509 MITM CA things do.
Autobanning always causes trouble if enough people use the server from one IP address.
> ufw enable
> ufw default deny
Of course after I do the server setup (which is usually a script that will change ssh ports, install packages, etc) I will allow other services in ufw.
Two hours later, when I read the mail, that account had already been compromised. I knew it was a risk, but had no idea that it would happen so __quickly__.
Change the root password to something long and complex.
And bam! Not even a full paragraph in and security fail. root login should be disabled completely, and all use of privileges should be through sudo. Debian sets this up for you automatically upon install if you supply an empty root password. Of course, disabling root is just the beginning (and the first user created needs to be locked down, as they are now essentially root).
I ended up popping out the drive and mounting it on another box, changed my fstab to noauto, got things to boot and reenabled root.
I figure distros that disable root are setup for these corner-cases where root is required.
The other obvious case is if you're using LDAP and the network is down. Maybe I haven't looked in the right places, but I haven't heard these things discussed.
Would you care to explain why?
Also, you can always sudo bash, or sudo su.
You have a point about that extra password. But not about SELinux and RBAC being created because of that, nor in comparing the security of a system with root to one where eveybody is root.
All said, I'm still unconvinced. The logging isn't that usefull (I've tried it), and the extra password isn't relevant enough. Also, none of them will detain a malicious user (Linux has very week defenses against you grabing your own password).
Not if you lock down sudo properly. If you're just doing "username ALL=(ALL) ALL", then yes, you've got a big gaping hole. Even for my main administrator accounts sudo is locked down to a specific list of commands. As for logging, I get an email every time sudo is run on my systems. It's not built into sudo, but there are multiple packages which take care of such things. You can even set it up to do remote logging to an external source.
As I understand RBAC, it was specifically created to break up the responsibilities of root to different roles with privileges which were then assigned based on different access control mechanisms.
Linux has very week defenses against you grabing your own password.
This sounds like a major hole which I'm sure the security community would love to hear about; care to elaborate? Just to pick one example, how does one grab a password entered over an ssh session?
We don't use a single deploy user though, instead, each user with deploy perms is in the sudo group.
Having a _shared_ account, with sudo privileges and a common password doesn't look smart.
Also, you're forcing devs to copy their public keys around.
I think you underestimate the benefits brought in by a centralized system, like LDAP, which would also allow you to manage the permissions with some more granularity
1) Many people seem to be recommending Puppet / Chef. How many servers or installs do you need before this is a good ROI? (Over using odd bash scripts or cPanel/WHM)
2) Am I right in thinking kernel updates don't get applied until the server is rebooted? If so, how / when do you manage this?
Really there's nothing stopping you from automating "some" machines and later pushing that out to the rest of your hosts. (Or just setting up trivial automation "configure NTP", "Disable root logins", "Append 'steve' to sudoers" - and keeping doing the rest by hand until you're confident.)
FWIW I use my toy configuration system, slaughter:
Ansible looks great. Much lighter weight but it can still scale up.
Puppet and chef would both benefit by supporting single run light weight modes of usage.
I use it to manage just 5 servers, with a Fabric script that rsyncs the manifests up to the servers then runs `puppet apply` to apply the changes.
and I see here I have capfiles and a system to bootstrap a new server. way too much stuff for this scale. (4 servers)
But you can setup a linux virtual server on linode and go through all of those steps on your own. Also test out your setup.
By the way, what the heck developers do on production systems?
As regards to unattended upgrades, again in small companies the choice is usually between them or no upgrades (no time for the test/deploy cycle across all platforms in use etc., less frequently used systems get forgotten and so on). As the article points out, its safest just to go for security updates only. If the security update breaks things, then yes, your system is down until you fix it. If a hacker breaches your system, your system is down until you re-deploy it, and then you have to deal with the PR fallout of lost credentials and data etc. Its only anecdotal, but I've had no problems with unattended security updates on my ubuntu boxes in the 5-ish years. Security breach attempts are a daily occurrence. It's nice if its not an either/or choice, but for many it is.
Seeing as he blocks SSH and only allows it from his IP, there won't be any SSH attempts reaching sshd at all, so fail2ban wouldn't be doing anything if it was looking at SSH only.
I have 30 different ips and fail2ban doesn't seem to ban them.
The hackers fail according to sshd but logwatch lists a list of chinese and russian attempts.
I was hoping my firewall would block them. I tried entering a block of ips but some of the same ones are connecting.
I don't know what this means:
Illegal users from: undef: 20 times 183.60.177.246: 7 times 217.14.134.68: 7 times 219.149.30.170: 6 times
Any pointers how to start? Good tutorials or walkthroughs?
IMO best practice is to firewall off everything except some bastion hosts or VPN gateway.
http://bsdly.blogspot.com/2013/02/theres-no-protection-in-hi...
I much prefer a deploy accound which DOES NOT have sudo access. I add firewall rules transparently redirecting 80/443 to non-privileged ports that the webapp is actually listening to. Hence no need to be root / sudo'ed for the deploy account.
You then could get a bit fancier and set the login shell for the deploy account to /bin/false or something fancy (like a fake honeypot shell) to add a bit of "defense in depth" in case someone exploiting a hole in your web server manages tries to drop down to a shell. You'd then use another account to do the (automated) deploy/start/stop/update/patch whatever.
I'd also say that during the first five minutes you should set the default firewalling rules to REJECT anything and then only whitelist what is actually allowed.
At which point your 'net connection blips and suddenly you wish you'd paid extra for console access.
One idea I had was to enable something annoying and kooky like port-knocking or OTP pass/connection enabled 'backdoor entrance' for emergencies, but ended up being too lazy, and realised it was just expanding the attack surface.
To my original point, we had an interesting setup for one network where the firewall changes were all manipulated via a script which required the change to be applied, and then confirmed after a short delay with a different command, otherwise after 5 minutes the ruleset would revert to the previous known-working one. It definitely saved some downtime/late night DC trips.
$ ssh -A -t office_user@office ssh production_user@production
I have the same private key in all my machines, so this worked transparently avoiding the IP restriction of iptables of the production machine.