The Reluctant Sysadmin's Guide to Securing a Linux Server
pboyd.io
pboyd.io
But as long as we're doing it anyway - the only thing that locking the root account gets you is assurance that if you ever bork the user you created in this guide (or sudo functionality as a whole) you'll have no way to recover without booting into another environment.
Perhaps one ought not take sysadmin advice from a blog post with a first sentence that reads "I’m not a sysadmin, and I don’t want to be".
AWS allows you to ssh into your hosts from within AWS. You just manage that security. NO ONE needs public ssh access, no one needs vpn ssh access just AWS ssh access. DON'T OVER COMPLICATE THINGS!
I agree with you. I am not gonna say don't follow a system engineers advice. I say follow everyone's advice but pick out the things that seem most reasonable. If it is extra work then you're doing it wrong, simplify everything so that the time spent on resolving issues is faster. Faster resolution means faster security fixes.
What you want to do is avoid fault, improve tolerance, but extend liability to the provider.
This is where your argument breaks down IMHO. Unless you are saying "don't expose port 22 to the world...", which is a common (small) part of security-in-depth.
> You also shift fault to AWS rather than your company and team. You did your diligence, you just have to access control and nothing more. IF AWS has a security breach that access to your systems completely on AWS and you can demand compensation.
This appears to be an instance of the "appeal to authority"[0] fallacy and is of little solace should server(s) one is responsible for become compromised.
This isn't fallacy it is reducing a businesses cognitive load.
My point exactly.
> You established what is supposed be a secure system yet that system failed due to provider security.
This makes no sense. By your own recommendation, "provider security" is an AWS offering.
> AWS is far more staffed than any other company why wouldn't you shift left to AWS? Why would you hire a fleet of security engineers to do what AWS already established?
What does Amazon's staffing have to do with best practices when securing a server deployed on their platform? Who said anything about "a fleet of security engineers"? How does any of that relate to securing that which one constructs, and ultimately is responsible for, when using said hosting services?
> You are breaking convention, reinventing the wheel and complicating an already simple system.
Are you saying that your original statement of "You also shift fault to AWS rather than your company and team" is somehow an accepted convention?
And what wheel did I "reinvent"?
Finally, was my identification of the common practice which is moving sshd off of port 22 the complication to which you refer?
https://aws.amazon.com/blogs/compute/new-using-amazon-ec2-in...
I really don't recommend extending responsibility for creature comforts. However you want to do this so be it.
> This makes no sense. By your own recommendation, "provider security" is an AWS offering.
Are you stating your system is infaliable? So why would you want to bear the infaliable claim and not shift it to the company providing it?
> What does Amazon's staffing have to do with best practices when securing a server deployed on their platform? Who said anything about "a fleet of security engineers"? How does any of that relate to securing that which one constructs, and ultimately is responsible for, when using said hosting services?
Tooling takes a team to support it. You think every company can afford a team to manage that tooling? And why should they? Not all businesses are tech companies but still need a digital footprint. They need to be selfconcious and choose provider practices to get the most out of them.
> Are you saying that your original statement of "You also shift fault to AWS rather than your company and team" is somehow an accepted convention? Providers own the responsibility of their technology. In terms of failure if access is correctly configured and managed, yet their technology fails they owe your businesses it is very simple.
> And what wheel did I "reinvent"?
Implementing old security practices. Why wouldn't you move to be better pratices and prevent larger holes in your network? Often companies get into this repetitious cycle of reimplementation or reinvention of existing tools and technology just to manage access especially ssh. The convention of using a cloud platform is to use a cloud platform's security access not some sketched up VPN and SSH system.
No, I am trying to remind you of the topic which was under discussion. To wit:
The Reluctant Sysadmin's Guide to Securing a Linux Server
A straw man fallacy (sometimes written as strawman) is the
informal fallacy of refuting an argument different from
the one actually under discussion, while not recognizing or
acknowledging the distinction.[0]
See above quote.
> You think ...
You do not know what I think nor my experiences, so please do not be so arrogant as to assume so.
>> And what wheel did I "reinvent"?
> Implementing old security practices.
Again, please refer to the *article under discussion*. In the event it remains unclear, I will restate its title:
The Reluctant Sysadmin's Guide to Securing a Linux Server
See previous strawman definition and link below.
> Often companies get into this repetitious cycle of reimplementation or reinvention of existing tools and technology just to manage access especially ssh. The convention of using a cloud platform is to use a cloud platform's security access not some sketched up VPN and SSH system.
Again, see previous strawman definition above and link below.
Note that the only ssh-related recommendation I proffered was:
Unless you are saying "don't expose port 22 to
the world...", which is a common (small) part of
security-in-depth.
As to "sketched up VPN and SSH system", I have no idea as to what you are referencing. Perhaps this is a recollection of a previous engagement wherein decisions made remind you of a bad situation similar to, but different than, this?
HTH
0 - https://en.wikipedia.org/wiki/Straw_man
EDIT: corrected spelling from "waas" to "was"
You would not believe how many companies are dependent on patching through users through VPNs in order to access remote hosts. I mean some have to because of no other solutions like managing their own on-prem. I kind of would be interested in AWS access management capable of being implemented within an on-prem.
If you rent compute space, then you trust them to responsibly use the hypervisor instead of snooping. If you trust that or not, you are all in and may as well cement over the external port 22.
Staffing count doesn't guarantee quality.
Multiple factors:
1FA: Password(1F) OR private key (password blank)(1F)
2FA: Private key(1F) with password(2F)
MFA: Private key(1F), w/ password(2F) AND OTP(3F)
Ssh has 2fa options if that's the real reason.
Fwiw, this guide also suggests setting up a wg connection which is no better than ssh, and probably worse in some ways.
https://docs.aws.amazon.com/systems-manager/latest/userguide...
Google Cloud has a similar gcloud compute ssh instance-name command, and I imagine there's a similar one on azure.
One could have a box with no public IP and no open ports and still use this to connect.
Via ssh? With an SSH key? Over port 22?
No, through their in-house proxy tools such as Session Manager or Identity Aware Proxy or whatever Azure has.
> With an SSH key?
Not at the edge, and not an SSH key you manage. A dynamically generated one managed by the cloud provider which exists just for that session. So, not really, not like you're thinking.
> Over port 22?
For the tunnel? No.
1. Install LTS server with a sudo admin user and complex password (yes, password, not key. I still end up using keys for backup automation because it's easier.)
2. apt install fail2ban. Don't even need to configure it: its default configuration will auto-ban any IPs trying to brute-force SSH.
3. apt install unattended-upgrades (nowadays installed by default)
I'd like to see anyone hack that.
You mention running containers with no elevated permissions. I've taken a liking to Docker because, in addition to its many benefits, it lets me separate MY processes from those of the OS. It's just so simple when I can identify the server's running services with "docker ps". I'm curious, when was the last time a Docker exploit could escape the container and modify the host? I read it happened in the early days of Docker but is that still a risk? They would need to: a) exploit a bug in a service you're relying on (eg nginx), b) use the former bug to exploit a bug in Docker, c) defeat the Linux kernel's namespace/cgroup isolation. Is that a realistic threat in 2023?
As opposed to borking the root user and being equally locked out? Assuming your sudo config is a "configure it once and then leave it forever" deal - which seems common IME - I can't see any way it would be different.
(Mind, this cuts both ways - once you force only key-based SSH, I generally don't see a problem with direct root access either.)
That's just perfect.
However, I would like to hear what someone who dislikes sysadmining but must anyway has to say. Why? A different perspective. I know some sysadmins who "left the car up on blocks," as in, they couldn't stop tweaking and fixing and so on, and they love that, and that's great.
Someone who is forced into the job, however, is communicating to me what I would expect to be the most necessary tasks. Maybe I also am in a similar place and would like to get on with the rest of my job. And in an Agile world, programmers get pushed into sysadmining ... at least, in some versions of Agile to which I have been subjected.
The only way a job is done consistently well is by those whom want to do that job well consistently.
HTH
EDIT: added missing "do" verb.
Sometimes a business can't or won't hire a real sysadmin. The people forced into the job need guides just as much.
> Sometimes a business can't or won't hire a real sysadmin. The people forced into the job need guides just as much.
I was making a more general statement regarding people having to do tasks they don't want to do, even if they have the skills to do them. Which makes resources helpful to "people forced into the job" rather difficult to produce.
How can one make a guide to assist people to be successful in something they don't want to do in the first place?
It is a vm so if you do something that would break sudo or all your users you should have a vm snapshot at your fingertips ready to restore from AWS interface.
Even if you are running bare metal you should setup snapshots first but nowadays no one runs bare metal web servers it is still som hyper visor with bunch of vm-s that are easy to backup restore or just delete and create fresh.
What?
I am the only admin on about a dozen machines. I'm not outsourcing that to a friend. That trust model is much more flawed than separation of duties and permissions segmentation as part of my administration routine.
If you are single person shop that is your choice, have a root account or whatever you feel like having.
> That's not true. It's not obvious what user you have that could do sudo. Thus it does improve security. I advice the same in my book (Deployment from Scratch) and I suggest that for both the host system and containers. There is little cost to not primarily using root.
It is true. To ensure root cannot be used when ssh'ing into a server, set "PermitRootLogin" to "no" in sshd_config (as mentioned in the OP).
Locking out root entirely, as further mentioned in the OP and suggested by your comment, does nothing to increase security regarding remote penetration attacks. Furthermore, should a non-root account which has sudo privileges be compromised, an argument could be made that having a functional root account with its own password accessible only locally and not enabling sudo is a more secure approach.
Either way, having a root account which can only be used locally ensures there is a recovery workflow should one be needed, as the GP enumerates.
That's not a unique or novel insight. For the case your system gets borked (either by yourself, your hardware or your cloud provider) you need a plan in advance:
1. How can I access the data the server has or how much of it can I afford to lose?
2. How do I get a replacement running within a time window acceptable for my usage?
The answers will be very different depending on your use case. But how you locked the root user has very little impact on them.
Booting into another environment is always one option in my plan so locking the root user doesn't frighten me.
As a dev, I say that's a good thing. I've administered my own systems for decades and helped in small startups where we had no full time admin so definitely not new to administering Linux.
For instance the chapter on web servers is far from today's best practices. It only mentions Apache http (nowadays Nginx is much more widespread), gives an advice about a default configuration which is no more default, and mentions a path that has changed in recent Debian installs. Even considering its age, the quality of this chapter is dubious: it forgets important points, like disabling .htaccess and directory listing, removing unused modules...
Modern tools are obviously missing from this guide: apparmor (though it was in use in 2017), nftables, systemd (unit settings that prevent /home access, prevent privilege escalation, etc)...
From the changelog, the document had one minor update in 2017 and one in 2013. It was mostly written in 2001-2007. Much has changed over the last 15 years.
> But I write software for the web
I'm going to guess it's a web server but it's just a guess.
What the traffic is seems irrelevant to 99.99% of servers out there, imo. Yes there's some questions of what deployments look like and what capabilities operators have but those are details outside the general concern of being safely online. IMO.
I'm going to counter that the overwhelming majority of hosts in existence do not, in fact, "sit on the internet".
Nope. Not by a long shot.
> What the traffic is seems irrelevant to 99.99% of servers out there, imo. Yes there's some questions of what deployments look like and what capabilities operators have but those are details outside the general concern of being safely online. IMO.
The following vulnerabilities listing just for the week of 2023-07-17 prove otherwise:
“ But I write software for the web, which means I’m never far from a server, and sometimes I’m the only one around.
So even if I didn’t want the job, I have it, and I need to take the security of these hosts seriously.”
Basic Linux server hardening is not a bad idea or skill to learn. Learning the basics manually help feed into understanding and using higher level solutions.
As long as you are using Debian, RHEL, Ubuntu, or CentOS implementing a basic minimum security baseline is as simple as following the CIS or DISA STIG guides for your OS.
There are plenty of scripts on GitHub that do this (AUDIT THEM FIRST), or you can even just use premade images from CIS in the cloud provider of your choice.
What the application is doing is relevant to application security, but the whole point of securing the OS is to eliminate the necessity for "trusting" the application.
When you are securing an operating system, you must assume the application that is exposed to the operating environment (be that the internet, local LAN, even simply user logged into the workstation in the case of a GUI or CLI app) is compromised.
The primary goal of most security measures is preventing and detecting privilege escalation and lateral movement within the OS or network.
There are a lot of best practices that apply in general to securing an operating system. If you want to dig deeper, one of the best resources for this information is provided by CIS (Center for Internet Security).
CIS has hardening standards for most OS's yes even including Ubuntu. https://www.cisecurity.org/benchmark/ubuntu_linux
These are standards that many security conscious organizations apply to their servers. The US government takes it a step further with DISA's STIGs.
DISA STIG's are similar to CIS's benchmarks, but result in an even more locked down environment and place extreme restrictions on which crypto libraries are allowed to be used.
In short, securing the OS is a standard best practice that all organizations should be doing. Unfortunately most startups lack engineers with the expertise in building custom linux images so a lot of folks are quite unfamiliar with hardening procedures.
You should absolutely NOT use a non-standard OS because you think it will be more secure. It's a much better idea to use known industry standard security benchmarks on supported Linux distributions than trying to bake your own standard some non Debian/RHEL based bistro.
Some of the line items can be a bit arcane or not as relevant in cloud environments, etc... but that's a far cry from calling them CYA.
Nothing cargo cult about enabling SE Linux, restricting access with IP tables, configuring AuditD and AIDE.
These are great ways to massively overcomplicate your system. Generally speaking, having encountered these tools, do not use them unless you're willing to dedicate about 2x the time you would otherwise spend administering the system.
Frankly you sound like the tired BOFH trope, if you don't see the benefits of security hardening I hope you are never responsible for anything important infrastructure wise in your organization.
Don't bother setting up a user account, use a public key authorized SSH session as root to do everything. Setting up UFW to block everything but what you should be serving is good. I don't see much point in things like Wireguard or this umask thing.
I don't think your situation has anything to do with what I described. It may still be linux, but it strikes me like saying that the maintenance manual is different between a sports car and a dump truck. Well yeah, obviously.
Bad though I think the original article might be, it would be 10x worse to attempt to write the reluctant sysadmin's guide to triple-digit workstation clusters in a university environment. Nothing about best practices for production web servers will apply for that, you need to hire an actual sysadmin.
If you're on Windows you can `wsl --install` and work with Linux (eg Ubuntu 2204).
You can also install Git Bash which comes with ssh and ssh-keygen.
Either way , same instructions.
in the case of a successful attack, some questions to ask could be:
- why did they manage to use wget?
- why {apache,nginx,postfix,exim,sendmail,...} is allowed to use wget, or curl, or nc or bash (or ...)?
- why is wget, curl, nc, telnet, .. installed on the server? can they be uninstalled? with (!!) if it's a container.
- why did they manage to execute files from /tmp, or /var/tmp, or /dev/shm? do these directories need write access for "others" or can they be mounted with "noexec"?
- ufw/iptables/nftables won't stop local binaries from opening outbound connections, how would you stop outbound connections by binary, path, etc?
- if they managed to wipe the logs, how could you have known all the commands they executed? could auditd+grafana (just an example) have helped here by sending logs to a remote server?
> ufw/iptables/nftables won't stop local binaries from opening outbound connections
Wait... Of course iptables/nftables can be used to prevent anything local from opening outbound connections. You can, say, easily have a firewall which only allows "NEW" traffic to be inbound traffic on either port 22 or 443.
They're called stateful firewalls for a reason.
For example on Debian you could configure the firewall so that the only user allowed to emit new traffic to get updates is the (/nonexistent:/user/sbin/nologin) user "_apt".
And for all those (not you) talking about the "cattle vs pet" thing, all this can be automated by hardening scripts you run exactly once, once you set up the server.
It's not because there are guides out there that every step in these guides have to be done manually, each time you configure a new server.
I did just install my first NixOS system so I’m indeed heading towards full automation.
But Tailscale is just a 3 sec process for any new server, 0 config needed, no holes in the firewall, nothing.
But I get the argument.
After that, as others have noted, create and review a threat model and use that to guide your hardening based on official guides:
https://www.debian.org/doc/manuals/securing-debian-manual/
and here's a readable introduction to the NIST STIGs:
https://cybergladius.com/nist-server-hardening-best-practice...
Over the last 3 years I have gone from being a timid junior web dev to reluctantly and hastily having to be the guy managing the Linux web servers and keeping the operation running and being hardened along the way.
On the one hand, the huge salary increase has been nice but on the other hand I am constantly thinking one day I'm gonna fuck it all up. I feel like I'm not doing this job any justice and that I'm way out of my element all the time.
I try to get better by reading blog posts like this and documentation and asking for advice but I just feel like an impostor all the time.
But employers are happy with the results and I guess that makes it tolerable.
So thanks for these types of guides!
https://www.digitalocean.com/community/tutorials/how-to-prot...
It's been trivially easy to prevent bots spamming basic SSH and HTTP attacks to every IPv4 address for a very long time.
"The best thing you can do is enable automated updates, and rely on your cloud provider's console for accessing the server and disabling all remote access otherwise."
1) I've run into to many issues letting the systems auto update. 2) Several times the AWS console (a web app so probably not as secure as a remote Linux ssh connection in my mind) failed to work, not just for me either, there have been multiple report I found on this, my remote ssh connection is the only way I could fix it since AWS doesn't have a remote serial console thing like Linode has (or had?).
https://aws.amazon.com/about-aws/whats-new/2021/03/introduci...
Maybe then look at the MITRE ATT&CK framework, Cyber Kill Chain etc.
I really hate to suggest them as it appears they have deviated in weirds ways from their original goal of protecting critical infrastructure from cybersecurity attacks, but CISA has many relevant documents.
DONT OVERCOMPLICATE WORK
Overcomplicating work means slower response times to solving problems.
The headline and post read like top shelf parody too, as if synthetically generated to summon we archetypes.
Is acl needed over, say, chown?
027 does a better job of keeping the model while tightening it up - worldly permissions are removed, users and groups are still meaningful.
This is what they're supplementing with ACLs, creating a frustrating problem of discovery by managing groups of users outside of groups
It's not necessarily wrong, I guess. There may be cases where someone wants this. ACLs are an answer, just not the one I'd suggest.
Why? Imagine 'Bob' leaves. Do you want to remove them from countless ACLs, or one group?
One is probably better off with 027, using groups, and focusing on SELinux or AppArmor. It will permit or deny things based on many things, including user context.
Bonus: it isn't limited to assets on disk. Things like gaining a shell and proxying services can be denied.
If you are sure you can use 077 without stuff breaking, awesome, but that's not always the case. Typically on systems using 077 you will find yourself using chmod a lot.
The reason that the author uses umask 077 and ACLs is, I think, just a mindset. By using 077, the file is restricted to only the owner, and the sysadmin does not need to think about group memberships. By extending read access using an ACL, this theme is continued; additional usernames will be appended as ACLs, but no group set of usernames needs to exist.
A file named "alfred" would, presumably, only ever needed to be read by root and alfred, but that's just the narrow case for the author's scheme.
But really the idea of being successful at anything while also having no knowledge of it is kind of a farcical contradiction.
It’s like saying, look, I just want to build a rocket engine. Tell me how without all the physics mumbo jumbo.
For me, I find the hardest part of securing systems are usually to decide what is good enough for the current situation.
No we don't. This creates problems with many packages. There's a reason for defaults and a reason not to follow cargo-culting security "recipes".
If you're a reluctant sysadmin that doesn't care, I'd recommend using Tailscale. It's wireguard without the drama, is extremely competent at piercing through almost any firewall [0], and has a great ACL system that lets you fine tune which accounts can access what.
It's also free (for now)!
Why not?
Also, setup auditd and rsyslog forwarding. Backup anything important.
So I'd argue it's impossible to build a correct threat model if all your vulnerabilities are expressed on code-level, rather than on "what software" or "what packages" are affected by it.